← All Test Cases

ALI-003

alias

medium

Repetitions

Documents

Questions

Reasoning

DIRECT

context-pack multi-query alias-expansion ops-operations

📖 In Plain English

What this category tests

Does query expansion match abbreviations to their full forms?

How the test works

A document is ingested using a full term ('bank reconciliation'). The query uses the abbreviation ('bank rec'). The brain's alias system must bridge the two.

Why it matters

Users mix formal and casual vocabulary — alias expansion is essential for natural querying.

Specifically for ALI-003

Tests brain_context_pack with query expansion — 'ops' should expand to 'operations' and find related docs.

⚙️ How a single rep runs

① Generate

Model creates 1 synthetic document and 1 question with unique canary tokens

→ Fresh content per run prevents memorization and proves real retrieval

② Ingest (MCP)

Model calls brain_ingest to store the 1 document

→ Tests the brain's storage and indexing pipeline

③ Query (MCP)

Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)

→ Core test — does the brain return correct evidence and let the model build a faithful answer?

④ Evaluate

Model judges the answer against ground truth (the document it generated in phase 1)

→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 3 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1510 chars

Generate a document using the FULL form of a term. Phase 3 uses brain_context_pack
with the abbreviation to test alias expansion via context pack queries.

Requirements:
- Choose a full term and abbreviation pair (vary per rep):
    "operations" / "ops"
    "configuration" / "config"
    "infrastructure" / "infra"
    "authentication" / "auth"
- Write 2-3 sentences about a fictional system using ONLY the full term, plus a canary token.
- source_id: KB-{{RUN}}-ALI-003-A-v1
- The question uses brain_context_pack with the abbreviation.

Output ONLY this JSON:
{
  "run_id": "{{RUN}}",
  "test_id": "ALI-003",
  "repetition": {{REP}},
  "documents": [
    {
      "source_id": "KB-{{RUN}}-ALI-003-A-v1",
      "content": "<2-3 sentences using the FULL term, plus canary>",
      "title": "<title using full term>",
      "version": 1
    }
  ],
  "questions": [
    "Call brain_context_pack with query='<abbreviation>' (e.g. query='ops'). Check whether the result includes content from the document about '<full term>'. Return the source_id and first sentence if found."
  ],
  "expected_answers": [
    {
      "question_index": 0,
      "correct_answer": "<first sentence of the document>",
      "required_source_ids": ["KB-{{RUN}}-ALI-003-A-v1"],
      "required_quote": "<first sentence>",
      "required_label": "DIRECT",
      "must_refuse": false,
      "notes": "brain_context_pack with the abbreviation should return results from the document about the full term via alias expansion."
    }
  ]
}

▶ ② Execution Instructions 276 chars

You have access to MCP Knowledge Brain tools.
Call brain_context_pack with query set to the abbreviation from the question.
Check whether the result bundle includes content from the expected document.

Question: {{QUESTION}}

Return ONLY this JSON:
{{REQUIRED_ANSWER_SCHEMA}}

⚖ ③ Judge Instructions 789 chars

Ground truth:
{{SETUP_CONTEXT_JSON}}

Tested AI answer:
{{EXEC_ANSWER_JSON}}

The AI called brain_context_pack with an abbreviation. Alias expansion should have included
the document that uses the full term in the returned context bundle.

Scoring:
- retrieval_tool_use (0-20): Did it call brain_context_pack with the abbreviation?
- evidence_fidelity (0-20): Correct source_id? Verbatim first sentence?
- reasoning_correctness (0-20): Was the document found in the context pack results?
- uncertainty_discipline (0-15): No fabricated content?
- output_format_compliance (0-15): Valid JSON?
- adversarial_resistance (0-10): Full marks.

Critical failures: searched with full term instead of abbreviation, document not found in context pack.

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

used_full_term_instead_of_abbreviation
document_not_in_context_pack_result

Recent Run History

3 runs

When	Run ID	Pass Rate	Avg Score	Reps
2026-05-24 13:08	20260524T130808Z-kqze	100%	85.0	1/1	View →
2026-05-24 12:41	20260524T124148Z-z2do	100%	100.0	1/1	View →
2026-05-24 11:37	20260524T113756Z-kduj	100%	100.0	1/1	View →

📄 Raw YAML cases/alias/ALI-003.yaml

schema_version: "1.0"
test_id: "ALI-003"
category: "alias"
severity: "medium"
repetitions: 3
reasoning_type: "DIRECT"
num_documents: 1
num_questions: 1
tags: ["context-pack", "multi-query", "alias-expansion", "ops-operations"]

setup_instructions: |
  Generate a document using the FULL form of a term. Phase 3 uses brain_context_pack
  with the abbreviation to test alias expansion via context pack queries.

  Requirements:
  - Choose a full term and abbreviation pair (vary per rep):
      "operations" / "ops"
      "configuration" / "config"
      "infrastructure" / "infra"
      "authentication" / "auth"
  - Write 2-3 sentences about a fictional system using ONLY the full term, plus a canary token.
  - source_id: KB-{{RUN}}-ALI-003-A-v1
  - The question uses brain_context_pack with the abbreviation.

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "ALI-003",
    "repetition": {{REP}},
    "documents": [
      {
        "source_id": "KB-{{RUN}}-ALI-003-A-v1",
        "content": "<2-3 sentences using the FULL term, plus canary>",
        "title": "<title using full term>",
        "version": 1
      }
    ],
    "questions": [
      "Call brain_context_pack with query='<abbreviation>' (e.g. query='ops'). Check whether the result includes content from the document about '<full term>'. Return the source_id and first sentence if found."
    ],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": "<first sentence of the document>",
        "required_source_ids": ["KB-{{RUN}}-ALI-003-A-v1"],
        "required_quote": "<first sentence>",
        "required_label": "DIRECT",
        "must_refuse": false,
        "notes": "brain_context_pack with the abbreviation should return results from the document about the full term via alias expansion."
      }
    ]
  }

execution_instructions: |
  You have access to MCP Knowledge Brain tools.
  Call brain_context_pack with query set to the abbreviation from the question.
  Check whether the result bundle includes content from the expected document.

  Question: {{QUESTION}}

  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  The AI called brain_context_pack with an abbreviation. Alias expansion should have included
  the document that uses the full term in the returned context bundle.

  Scoring:
  - retrieval_tool_use (0-20): Did it call brain_context_pack with the abbreviation?
  - evidence_fidelity (0-20): Correct source_id? Verbatim first sentence?
  - reasoning_correctness (0-20): Was the document found in the context pack results?
  - uncertainty_discipline (0-15): No fabricated content?
  - output_format_compliance (0-15): Valid JSON?
  - adversarial_resistance (0-10): Full marks.

  Critical failures: searched with full term instead of abbreviation, document not found in context pack.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "used_full_term_instead_of_abbreviation"
  - "document_not_in_context_pack_result"