← All Test Cases

NIH-001

needle haystack

high

Repetitions

Documents

Questions

Reasoning

DIRECT

retrieval-precision distractor-load needle-in-haystack

📖 In Plain English

What this category tests

Can the brain find one specific document among many distractors?

How the test works

One target document with a unique canary token is ingested alongside 10+ similar-sounding distractor documents. The test uses the canary to find only the target.

Why it matters

Precision matters at scale — a brain that returns 'mostly relevant' results is unusable in production.

Specifically for NIH-001

Needle-in-haystack with 11 documents (1 target + 10 distractors) — canary token must find only the target.

⚙️ How a single rep runs

① Generate

Model creates 11 synthetic documents and 1 question with unique canary tokens

→ Fresh content per run prevents memorization and proves real retrieval

② Ingest (MCP)

Model calls brain_ingest to store the 11 documents

→ Tests the brain's storage and indexing pipeline

③ Query (MCP)

Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)

→ Core test — does the brain return correct evidence and let the model build a faithful answer?

④ Evaluate

Model judges the answer against ground truth (the document it generated in phase 1)

→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 3 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1635 chars

Generate ONE target document containing a specific unique fact, then generate TEN distractor
documents about similar-sounding but different topics. Ingest all 11 documents.

Requirements:
- Target document: one sentence with a unique fictional fact and a canary token.
  Example: "The obsidian compass belongs to Navigator Elric. Canary: teal-rook-291."
- 10 distractor documents: each about a different fictional compass, navigator, or object.
  They must NOT mention the canary or the exact combination in the target.
- source_ids: KB-{{RUN}}-NIH-001-TARGET-v1 for target, KB-{{RUN}}-NIH-001-D01-v1 through D10-v1 for distractors.
- Question: use the canary to ask for the exact fact from the target document only.

Output ONLY this JSON:
{
  "run_id": "{{RUN}}",
  "test_id": "NIH-001",
  "repetition": {{REP}},
  "documents": [
    {"source_id": "KB-{{RUN}}-NIH-001-TARGET-v1", "content": "<target sentence with canary>", "title": "<title>", "version": 1},
    {"source_id": "KB-{{RUN}}-NIH-001-D01-v1", "content": "<distractor 1>", "title": "<title>", "version": 1},
    ... (10 distractors total)
  ],
  "questions": ["Using canary '<token>', find the exact fact from the target document and return the source_id and verbatim sentence."],
  "expected_answers": [{
    "question_index": 0,
    "correct_answer": "<the target fact>",
    "required_source_ids": ["KB-{{RUN}}-NIH-001-TARGET-v1"],
    "required_quote": "<exact target sentence>",
    "required_label": "DIRECT",
    "must_refuse": false,
    "notes": "Must find the ONE target doc among 10 distractors using the canary. Any distractor source_id is a failure."
  }]
}

▶ ② Execution Instructions 543 chars

You have access to MCP Knowledge Brain tools.

Use brain_search with the exact canary token from the question.
The canary is unique — only ONE document contains it.
Find that document. Return immediately once found — do not search further.

⚠️ OUTPUT RULE: As soon as you find the target document, stop reasoning
and output the JSON. Do not re-examine other documents. Do not verify.
Just output the required JSON schema with the found document's source_id and quote.

Question: {{QUESTION}}

Return ONLY this JSON:
{{REQUIRED_ANSWER_SCHEMA}}

⚖ ③ Judge Instructions 355 chars

Ground truth:
{{SETUP_CONTEXT_JSON}}

Tested AI answer:
{{EXEC_ANSWER_JSON}}

The target document was ONE of 11 ingested documents. The AI must find exactly the target using the canary.
Critical failures: cited any distractor source_id instead of the target, hallucinated source_id, cited multiple sources.

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

cited_distractor_instead_of_target
hallucinated_source_id
wrong_final_answer

Recent Run History

2 runs

When	Run ID	Pass Rate	Avg Score	Reps
2026-05-24 13:08	20260524T130808Z-kqze	0%	65.0	0/1	View →
2026-05-24 11:37	20260524T113756Z-kduj	100%	100.0	1/1	View →

📄 Raw YAML cases/needle_haystack/NIH-001.yaml

schema_version: "1.0"
test_id: "NIH-001"
category: "needle_haystack"
severity: "high"
repetitions: 3
reasoning_type: "DIRECT"
num_documents: 11
num_questions: 1
tags: ["retrieval-precision", "distractor-load", "needle-in-haystack"]

setup_instructions: |
  Generate ONE target document containing a specific unique fact, then generate TEN distractor
  documents about similar-sounding but different topics. Ingest all 11 documents.

  Requirements:
  - Target document: one sentence with a unique fictional fact and a canary token.
    Example: "The obsidian compass belongs to Navigator Elric. Canary: teal-rook-291."
  - 10 distractor documents: each about a different fictional compass, navigator, or object.
    They must NOT mention the canary or the exact combination in the target.
  - source_ids: KB-{{RUN}}-NIH-001-TARGET-v1 for target, KB-{{RUN}}-NIH-001-D01-v1 through D10-v1 for distractors.
  - Question: use the canary to ask for the exact fact from the target document only.

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "NIH-001",
    "repetition": {{REP}},
    "documents": [
      {"source_id": "KB-{{RUN}}-NIH-001-TARGET-v1", "content": "<target sentence with canary>", "title": "<title>", "version": 1},
      {"source_id": "KB-{{RUN}}-NIH-001-D01-v1", "content": "<distractor 1>", "title": "<title>", "version": 1},
      ... (10 distractors total)
    ],
    "questions": ["Using canary '<token>', find the exact fact from the target document and return the source_id and verbatim sentence."],
    "expected_answers": [{
      "question_index": 0,
      "correct_answer": "<the target fact>",
      "required_source_ids": ["KB-{{RUN}}-NIH-001-TARGET-v1"],
      "required_quote": "<exact target sentence>",
      "required_label": "DIRECT",
      "must_refuse": false,
      "notes": "Must find the ONE target doc among 10 distractors using the canary. Any distractor source_id is a failure."
    }]
  }

execution_instructions: |
  You have access to MCP Knowledge Brain tools.

  Use brain_search with the exact canary token from the question.
  The canary is unique — only ONE document contains it.
  Find that document. Return immediately once found — do not search further.

  ⚠️ OUTPUT RULE: As soon as you find the target document, stop reasoning
  and output the JSON. Do not re-examine other documents. Do not verify.
  Just output the required JSON schema with the found document's source_id and quote.

  Question: {{QUESTION}}

  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  The target document was ONE of 11 ingested documents. The AI must find exactly the target using the canary.
  Critical failures: cited any distractor source_id instead of the target, hallucinated source_id, cited multiple sources.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "cited_distractor_instead_of_target"
  - "hallucinated_source_id"
  - "wrong_final_answer"