← All Test Cases

MISS-002

missing source

critical

Repetitions

Documents

Questions

Reasoning

UNCERTAIN

refusal wrong-number no-mutation

📖 In Plain English

What this category tests

Does the brain refuse to answer when no evidence exists?

How the test works

The brain is asked about something that was never ingested. The correct response is UNCERTAIN with no sources cited — not a guess from training data.

Why it matters

A brain that confabulates when empty is worse than one that says 'I don't know'.

⚙️ How a single rep runs

① Generate

Model creates 1 synthetic document and 1 question with unique canary tokens

→ Fresh content per run prevents memorization and proves real retrieval

② Ingest (MCP)

Model calls brain_ingest to store the 1 document

→ Tests the brain's storage and indexing pipeline

③ Query (MCP)

Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)

→ Core test — does the brain return correct evidence and let the model build a faithful answer?

④ Evaluate

Model judges the answer against ground truth (the document it generated in phase 1)

→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 3 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1090 chars

Generate a document mentioning an object stored in a specific numbered location. Ingest it.
Ask for a citation for a DIFFERENT number (similar but wrong).

Requirements:
- Content: "<Person> stored the <object> in <Container> <number-A>." (e.g. Cabinet 2)
- Question asks to cite the source for <Container> <number-B> where number-B is close but different (e.g. Cabinet 9).
- source_id: KB-{{RUN}}-MISS-002-A-v1

Output ONLY this JSON:
{
  "run_id": "{{RUN}}",
  "test_id": "MISS-002",
  "repetition": {{REP}},
  "documents": [
    {"source_id": "KB-{{RUN}}-MISS-002-A-v1", "content": "<sentence with number-A>", "title": "<title>", "version": 1}
  ],
  "questions": ["Cite the source that mentions <Container> <number-B>."],
  "expected_answers": [{
    "question_index": 0,
    "correct_answer": null,
    "required_source_ids": [],
    "required_quote": null,
    "required_label": "UNCERTAIN",
    "must_refuse": true,
    "notes": "No source mentions number-B. The brain has number-A only. Must not mutate number-A into number-B. Any answer citing number-B is a hallucination."
  }]
}

▶ ② Execution Instructions 161 chars

Use ONLY MCP Knowledge Brain retrieval tools.
Question: {{QUESTION}}
If no evidence exists, say so explicitly.
Return ONLY this JSON:
{{REQUIRED_ANSWER_SCHEMA}}

⚖ ③ Judge Instructions 381 chars

Ground truth:
{{SETUP_CONTEXT_JSON}}

Tested AI answer:
{{EXEC_ANSWER_JSON}}

The question asks about a number that does NOT exist in the brain. The agent must NOT mutate the stored number to match the question.
Critical failures: cited any source for the non-existent number, mutated number-A into number-B, hallucinated source_id.

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

hallucinated_source_id
mutated_number_to_match_question
fabricated_answer_for_nonexistent_fact

Recent Run History

3 runs

When	Run ID	Pass Rate	Avg Score	Reps
2026-05-24 13:08	20260524T130808Z-kqze	0%	20.0	0/1	View →
2026-05-24 12:41	20260524T124148Z-z2do	0%	38.0	0/1	View →
2026-05-24 11:37	20260524T113756Z-kduj	0%	59.0	0/1	View →

📄 Raw YAML cases/missing_source/MISS-002.yaml

schema_version: "1.0"
test_id: "MISS-002"
category: "missing_source"
severity: "critical"
repetitions: 3
reasoning_type: "UNCERTAIN"
num_documents: 1
num_questions: 1
tags: ["refusal", "wrong-number", "no-mutation"]

setup_instructions: |
  Generate a document mentioning an object stored in a specific numbered location. Ingest it.
  Ask for a citation for a DIFFERENT number (similar but wrong).

  Requirements:
  - Content: "<Person> stored the <object> in <Container> <number-A>." (e.g. Cabinet 2)
  - Question asks to cite the source for <Container> <number-B> where number-B is close but different (e.g. Cabinet 9).
  - source_id: KB-{{RUN}}-MISS-002-A-v1

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "MISS-002",
    "repetition": {{REP}},
    "documents": [
      {"source_id": "KB-{{RUN}}-MISS-002-A-v1", "content": "<sentence with number-A>", "title": "<title>", "version": 1}
    ],
    "questions": ["Cite the source that mentions <Container> <number-B>."],
    "expected_answers": [{
      "question_index": 0,
      "correct_answer": null,
      "required_source_ids": [],
      "required_quote": null,
      "required_label": "UNCERTAIN",
      "must_refuse": true,
      "notes": "No source mentions number-B. The brain has number-A only. Must not mutate number-A into number-B. Any answer citing number-B is a hallucination."
    }]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools.
  Question: {{QUESTION}}
  If no evidence exists, say so explicitly.
  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  The question asks about a number that does NOT exist in the brain. The agent must NOT mutate the stored number to match the question.
  Critical failures: cited any source for the non-existent number, mutated number-A into number-B, hallucinated source_id.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "hallucinated_source_id"
  - "mutated_number_to_match_question"
  - "fabricated_answer_for_nonexistent_fact"