← All Test Cases

ALI-002

alias

high

Repetitions

Documents

Questions

Reasoning

DIRECT

brain-update-aliases dynamic-alias alias-creation query-expansion

📖 In Plain English

What this category tests

Does query expansion match abbreviations to their full forms?

How the test works

A document is ingested using a full term ('bank reconciliation'). The query uses the abbreviation ('bank rec'). The brain's alias system must bridge the two.

Why it matters

Users mix formal and casual vocabulary — alias expansion is essential for natural querying.

⚙️ How a single rep runs

① Generate

Model creates 1 synthetic document and 1 question with unique canary tokens

→ Fresh content per run prevents memorization and proves real retrieval

② Ingest (MCP)

Model calls brain_ingest to store the 1 document

→ Tests the brain's storage and indexing pipeline

③ Query (MCP)

Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)

→ Core test — does the brain return correct evidence and let the model build a faithful answer?

④ Evaluate

Model judges the answer against ground truth (the document it generated in phase 1)

→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 3 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1517 chars

Generate a document using a unique fictional term. Phase 3 will:
1. Add a NEW alias for that term using brain_update_aliases
2. Immediately search using that new alias

Requirements:
- Invent a fictional technical term (e.g. "chromatic oscillation") and a short abbreviation
  or nickname for it (e.g. "chro-osc") that is NOT a real word.
- Write 2-3 sentences about this term including a canary token.
- source_id: KB-{{RUN}}-ALI-002-A-v1
- The question instructs the model to add the alias and then search with it.

Output ONLY this JSON:
{
  "run_id": "{{RUN}}",
  "test_id": "ALI-002",
  "repetition": {{REP}},
  "documents": [
    {
      "source_id": "KB-{{RUN}}-ALI-002-A-v1",
      "content": "<2-3 sentences about the fictional term, with canary>",
      "title": "<title>",
      "version": 1
    }
  ],
  "questions": [
    "Step 1: Call brain_update_aliases to add '<abbreviation>' as an alias for '<full fictional term>'. Step 2: Immediately search using only the abbreviation '<abbreviation>'. Return the source_id and first sentence of any document found."
  ],
  "expected_answers": [
    {
      "question_index": 0,
      "correct_answer": "<first sentence of the document>",
      "required_source_ids": ["KB-{{RUN}}-ALI-002-A-v1"],
      "required_quote": "<first sentence>",
      "required_label": "DIRECT",
      "must_refuse": false,
      "notes": "The alias must take effect immediately after brain_update_aliases is called. Searching with the new alias must find the document."
    }
  ]
}

▶ ② Execution Instructions 364 chars

You have access to MCP Knowledge Brain tools including brain_update_aliases.

Follow the two steps in the question exactly:
1. Call brain_update_aliases with the add parameter mapping the abbreviation to the full term.
2. Call brain_search using ONLY the abbreviation (not the full term).

Question: {{QUESTION}}

Return ONLY this JSON:
{{REQUIRED_ANSWER_SCHEMA}}

⚖ ③ Judge Instructions 827 chars

Ground truth:
{{SETUP_CONTEXT_JSON}}

Tested AI answer:
{{EXEC_ANSWER_JSON}}

The AI added a new alias using brain_update_aliases and then immediately searched with that alias.
The alias must work immediately — no delay or restart required.

Scoring:
- retrieval_tool_use (0-20): Did it call brain_update_aliases first, then brain_search with the alias?
- evidence_fidelity (0-20): Correct source_id? Verbatim quote?
- reasoning_correctness (0-20): Was the document found using the newly created alias?
- uncertainty_discipline (0-15): No fabricated content?
- output_format_compliance (0-15): Valid JSON?
- adversarial_resistance (0-10): Full marks.

Critical failures: skipped brain_update_aliases, searched full term instead of alias, document not found after alias creation.

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

skipped_update_aliases_call
searched_full_term_instead_of_alias
alias_did_not_work_immediately

Recent Run History

3 runs

When	Run ID	Pass Rate	Avg Score	Reps
2026-05-24 13:08	20260524T130808Z-kqze	100%	85.0	1/1	View →
2026-05-24 12:41	20260524T124148Z-z2do	100%	100.0	1/1	View →
2026-05-24 11:37	20260524T113756Z-kduj	100%	99.0	1/1	View →

📄 Raw YAML cases/alias/ALI-002.yaml

schema_version: "1.0"
test_id: "ALI-002"
category: "alias"
severity: "high"
repetitions: 3
reasoning_type: "DIRECT"
num_documents: 1
num_questions: 1
tags: ["brain-update-aliases", "dynamic-alias", "alias-creation", "query-expansion"]

setup_instructions: |
  Generate a document using a unique fictional term. Phase 3 will:
  1. Add a NEW alias for that term using brain_update_aliases
  2. Immediately search using that new alias

  Requirements:
  - Invent a fictional technical term (e.g. "chromatic oscillation") and a short abbreviation
    or nickname for it (e.g. "chro-osc") that is NOT a real word.
  - Write 2-3 sentences about this term including a canary token.
  - source_id: KB-{{RUN}}-ALI-002-A-v1
  - The question instructs the model to add the alias and then search with it.

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "ALI-002",
    "repetition": {{REP}},
    "documents": [
      {
        "source_id": "KB-{{RUN}}-ALI-002-A-v1",
        "content": "<2-3 sentences about the fictional term, with canary>",
        "title": "<title>",
        "version": 1
      }
    ],
    "questions": [
      "Step 1: Call brain_update_aliases to add '<abbreviation>' as an alias for '<full fictional term>'. Step 2: Immediately search using only the abbreviation '<abbreviation>'. Return the source_id and first sentence of any document found."
    ],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": "<first sentence of the document>",
        "required_source_ids": ["KB-{{RUN}}-ALI-002-A-v1"],
        "required_quote": "<first sentence>",
        "required_label": "DIRECT",
        "must_refuse": false,
        "notes": "The alias must take effect immediately after brain_update_aliases is called. Searching with the new alias must find the document."
      }
    ]
  }

execution_instructions: |
  You have access to MCP Knowledge Brain tools including brain_update_aliases.

  Follow the two steps in the question exactly:
  1. Call brain_update_aliases with the add parameter mapping the abbreviation to the full term.
  2. Call brain_search using ONLY the abbreviation (not the full term).

  Question: {{QUESTION}}

  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  The AI added a new alias using brain_update_aliases and then immediately searched with that alias.
  The alias must work immediately — no delay or restart required.

  Scoring:
  - retrieval_tool_use (0-20): Did it call brain_update_aliases first, then brain_search with the alias?
  - evidence_fidelity (0-20): Correct source_id? Verbatim quote?
  - reasoning_correctness (0-20): Was the document found using the newly created alias?
  - uncertainty_discipline (0-15): No fabricated content?
  - output_format_compliance (0-15): Valid JSON?
  - adversarial_resistance (0-10): Full marks.

  Critical failures: skipped brain_update_aliases, searched full term instead of alias, document not found after alias creation.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "skipped_update_aliases_call"
  - "searched_full_term_instead_of_alias"
  - "alias_did_not_work_immediately"