← All Test Cases

IDN-001

identifier

critical

Repetitions

Documents

Questions

Reasoning

DIRECT

exact-token canary identifier zero-tolerance

📖 In Plain English

What this category tests

Can the brain retrieve documents by exact identifiers (canaries, IDs, structured tokens)?

How the test works

Documents contain unique identifiers like 'TRK-9182' or 'azure-hawk-419'. The test queries with the exact identifier and expects the right document back.

Why it matters

Identifier-based lookup must be 100% reliable — it's how systems reference specific records.

⚙️ How a single rep runs

① Generate

Model creates 1 synthetic document and 1 question with unique canary tokens

→ Fresh content per run prevents memorization and proves real retrieval

② Ingest (MCP)

Model calls brain_ingest to store the 1 document

→ Tests the brain's storage and indexing pipeline

③ Query (MCP)

Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)

→ Core test — does the brain return correct evidence and let the model build a faithful answer?

④ Evaluate

Model judges the answer against ground truth (the document it generated in phase 1)

→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 3 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1400 chars

Generate one synthetic document containing a unique 3-word hyphenated canary token and ingest it.

Requirements:
- Invent a fictional object, a fictional location, and a unique canary token (e.g. "azure-hawk-419").
- Document content: ONE sentence combining object, location, and canary, e.g.:
  "The copper lantern rests in Chamber Veltris-88 [canary: azure-hawk-419]."
- source_id: KB-{{RUN}}-IDN-001-A-v1
- Question must ask to find the document using ONLY the canary token (no object or location mentioned).

Output ONLY this JSON:
{
  "run_id": "{{RUN}}",
  "test_id": "IDN-001",
  "repetition": {{REP}},
  "documents": [
    {
      "source_id": "KB-{{RUN}}-IDN-001-A-v1",
      "content": "<single sentence with object, location, canary>",
      "title": "<short title>",
      "version": 1
    }
  ],
  "questions": [
    "Search the brain for the exact canary token '<token>'. Return the source_id and the verbatim document sentence."
  ],
  "expected_answers": [
    {
      "question_index": 0,
      "correct_answer": "<the document sentence — exact copy>",
      "required_source_ids": ["KB-{{RUN}}-IDN-001-A-v1"],
      "required_quote": "<exact copy of document content>",
      "required_label": "DIRECT",
      "must_refuse": false,
      "notes": "Zero tolerance: the canary token must be found exactly. Any hallucinated source_id or wrong quote is a critical failure."
    }
  ]
}

▶ ② Execution Instructions 223 chars

You have access to MCP Knowledge Brain retrieval tools.
Use brain_search with the exact canary token from the question. Do not guess or use memory.

Question: {{QUESTION}}

Return ONLY this JSON:
{{REQUIRED_ANSWER_SCHEMA}}

⚖ ③ Judge Instructions 804 chars

Ground truth:
{{SETUP_CONTEXT_JSON}}

Tested AI answer:
{{EXEC_ANSWER_JSON}}

The AI must find a document using only its canary token. This is an exact-match identifier retrieval test.
Zero tolerance for wrong source_id or inexact quote.

Scoring:
- retrieval_tool_use (0-20): Did it call brain_search with the canary token?
- evidence_fidelity (0-20): Is the source_id exactly KB-{{RUN}}-IDN-001-A-v1? Is the quote byte-for-byte exact?
- reasoning_correctness (0-20): Is the answer the full document sentence?
- uncertainty_discipline (0-15): Did it avoid guessing?
- output_format_compliance (0-15): Valid JSON?
- adversarial_resistance (0-10): Full marks (not adversarial).

Critical failures: wrong/hallucinated source_id, inexact quote, wrong answer.

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

hallucinated_source_id
wrong_final_answer
quote_not_verbatim

Recent Run History

3 runs

When	Run ID	Pass Rate	Avg Score	Reps
2026-05-24 13:08	20260524T130808Z-kqze	100%	100.0	1/1	View →
2026-05-24 12:41	20260524T124148Z-z2do	100%	100.0	1/1	View →
2026-05-24 11:37	20260524T113756Z-kduj	100%	100.0	1/1	View →

📄 Raw YAML cases/identifier/IDN-001.yaml

schema_version: "1.0"
test_id: "IDN-001"
category: "identifier"
severity: "critical"
repetitions: 3
reasoning_type: "DIRECT"
num_documents: 1
num_questions: 1
tags: ["exact-token", "canary", "identifier", "zero-tolerance"]

setup_instructions: |
  Generate one synthetic document containing a unique 3-word hyphenated canary token and ingest it.

  Requirements:
  - Invent a fictional object, a fictional location, and a unique canary token (e.g. "azure-hawk-419").
  - Document content: ONE sentence combining object, location, and canary, e.g.:
    "The copper lantern rests in Chamber Veltris-88 [canary: azure-hawk-419]."
  - source_id: KB-{{RUN}}-IDN-001-A-v1
  - Question must ask to find the document using ONLY the canary token (no object or location mentioned).

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "IDN-001",
    "repetition": {{REP}},
    "documents": [
      {
        "source_id": "KB-{{RUN}}-IDN-001-A-v1",
        "content": "<single sentence with object, location, canary>",
        "title": "<short title>",
        "version": 1
      }
    ],
    "questions": [
      "Search the brain for the exact canary token '<token>'. Return the source_id and the verbatim document sentence."
    ],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": "<the document sentence — exact copy>",
        "required_source_ids": ["KB-{{RUN}}-IDN-001-A-v1"],
        "required_quote": "<exact copy of document content>",
        "required_label": "DIRECT",
        "must_refuse": false,
        "notes": "Zero tolerance: the canary token must be found exactly. Any hallucinated source_id or wrong quote is a critical failure."
      }
    ]
  }

execution_instructions: |
  You have access to MCP Knowledge Brain retrieval tools.
  Use brain_search with the exact canary token from the question. Do not guess or use memory.

  Question: {{QUESTION}}

  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  The AI must find a document using only its canary token. This is an exact-match identifier retrieval test.
  Zero tolerance for wrong source_id or inexact quote.

  Scoring:
  - retrieval_tool_use (0-20): Did it call brain_search with the canary token?
  - evidence_fidelity (0-20): Is the source_id exactly KB-{{RUN}}-IDN-001-A-v1? Is the quote byte-for-byte exact?
  - reasoning_correctness (0-20): Is the answer the full document sentence?
  - uncertainty_discipline (0-15): Did it avoid guessing?
  - output_format_compliance (0-15): Valid JSON?
  - adversarial_resistance (0-10): Full marks (not adversarial).

  Critical failures: wrong/hallucinated source_id, inexact quote, wrong answer.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "hallucinated_source_id"
  - "wrong_final_answer"
  - "quote_not_verbatim"