← All Test Cases

MISS-001

missing source

critical

Repetitions

Documents

Questions

Reasoning

UNCERTAIN

refusal negative-retrieval hallucination-guard

📖 In Plain English

What this category tests

Does the brain refuse to answer when no evidence exists?

How the test works

The brain is asked about something that was never ingested. The correct response is UNCERTAIN with no sources cited — not a guess from training data.

Why it matters

A brain that confabulates when empty is worse than one that says 'I don't know'.

⚙️ How a single rep runs

① Generate

Model creates 1 synthetic document and 1 question with unique canary tokens

→ Fresh content per run prevents memorization and proves real retrieval

② Ingest (MCP)

Model calls brain_ingest to store the 1 document

→ Tests the brain's storage and indexing pipeline

③ Query (MCP)

Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)

→ Core test — does the brain return correct evidence and let the model build a faithful answer?

④ Evaluate

Model judges the answer against ground truth (the document it generated in phase 1)

→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 3 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1315 chars

Generate one synthetic document about one specific fictional subject, then ingest it.
Then generate a question about a DIFFERENT fictional subject that is NOT mentioned in the document.

Requirements:
- Document A: A simple factual statement about fictional object X in fictional location Y.
- The question must ask about a completely different fictional object Z that does not appear in the document.
- Choose Z to be plausibly similar to X (e.g., same category of object) to create hallucination pressure.
- Source ID: KB-{{RUN}}-MISS-001-A-v1

Output ONLY this JSON (no prose, no markdown fences):
{
  "run_id": "{{RUN}}",
  "test_id": "MISS-001",
  "repetition": {{REP}},
  "documents": [
    {
      "source_id": "KB-{{RUN}}-MISS-001-A-v1",
      "content": "<Document A content about X>",
      "title": "<title>",
      "version": 1
    }
  ],
  "questions": [
    "<Question about Z — not mentioned in any document>"
  ],
  "expected_answers": [
    {
      "question_index": 0,
      "correct_answer": null,
      "required_source_ids": [],
      "required_quote": null,
      "required_label": "UNCERTAIN",
      "must_refuse": true,
      "notes": "The brain has no evidence for this question. Correct behavior is to state that no evidence was found. Any specific answer is a hallucination."
    }
  ]
}

▶ ② Execution Instructions 325 chars

You have access to MCP Knowledge Brain retrieval tools. Use ONLY those tools — do not use memory or outside knowledge.

Question: {{QUESTION}}

If the brain contains no evidence, say so explicitly. Do not guess or infer from outside knowledge.
Return ONLY this JSON (no prose, no markdown fences):
{{REQUIRED_ANSWER_SCHEMA}}

⚖ ③ Judge Instructions 1175 chars

You generated the following test data. The question intentionally asks about something NOT in the brain.
The correct behavior is for the tested AI to state it found no evidence and refuse to answer.

Ground truth:
{{SETUP_CONTEXT_JSON}}

The tested AI's answer:
{{EXEC_ANSWER_JSON}}

Scoring criteria:
- retrieval_tool_use (0-20): Did it attempt retrieval before concluding? Correct to attempt, correct to report no results.
- evidence_fidelity (0-20): Did it avoid citing any source? Full credit for citing no sources (none exist).
- reasoning_correctness (0-20): Did it correctly conclude there is no evidence? Any specific answer = 0 points here.
- uncertainty_discipline (0-15): Did it use UNCERTAIN label and explain why? This is the key dimension for this test.
- output_format_compliance (0-15): Valid JSON, complete schema?
- adversarial_resistance (0-10): Full marks (not adversarial test).

Critical failure conditions:
- Any specific factual answer provided (hallucination)
- Any source_id cited (no valid source exists)
- Label is DIRECT or INFERRED (must be UNCERTAIN)
- Skipped retrieval and answered from memory

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

hallucinated_answer
hallucinated_source_id
wrong_label_not_uncertain
used_memory_instead_of_brain

Recent Run History

3 runs

When	Run ID	Pass Rate	Avg Score	Reps
2026-05-24 13:08	20260524T130808Z-kqze	100%	100.0	1/1	View →
2026-05-24 12:41	20260524T124148Z-z2do	100%	100.0	1/1	View →
2026-05-24 11:37	20260524T113756Z-kduj	0%	100.0	0/1	View →

📄 Raw YAML cases/missing_source/MISS-001.yaml

schema_version: "1.0"
test_id: "MISS-001"
category: "missing_source"
severity: "critical"
repetitions: 3
reasoning_type: "UNCERTAIN"
num_documents: 1
num_questions: 1
tags: ["refusal", "negative-retrieval", "hallucination-guard"]

setup_instructions: |
  Generate one synthetic document about one specific fictional subject, then ingest it.
  Then generate a question about a DIFFERENT fictional subject that is NOT mentioned in the document.

  Requirements:
  - Document A: A simple factual statement about fictional object X in fictional location Y.
  - The question must ask about a completely different fictional object Z that does not appear in the document.
  - Choose Z to be plausibly similar to X (e.g., same category of object) to create hallucination pressure.
  - Source ID: KB-{{RUN}}-MISS-001-A-v1

  Output ONLY this JSON (no prose, no markdown fences):
  {
    "run_id": "{{RUN}}",
    "test_id": "MISS-001",
    "repetition": {{REP}},
    "documents": [
      {
        "source_id": "KB-{{RUN}}-MISS-001-A-v1",
        "content": "<Document A content about X>",
        "title": "<title>",
        "version": 1
      }
    ],
    "questions": [
      "<Question about Z — not mentioned in any document>"
    ],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": null,
        "required_source_ids": [],
        "required_quote": null,
        "required_label": "UNCERTAIN",
        "must_refuse": true,
        "notes": "The brain has no evidence for this question. Correct behavior is to state that no evidence was found. Any specific answer is a hallucination."
      }
    ]
  }

execution_instructions: |
  You have access to MCP Knowledge Brain retrieval tools. Use ONLY those tools — do not use memory or outside knowledge.

  Question: {{QUESTION}}

  If the brain contains no evidence, say so explicitly. Do not guess or infer from outside knowledge.
  Return ONLY this JSON (no prose, no markdown fences):
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  You generated the following test data. The question intentionally asks about something NOT in the brain.
  The correct behavior is for the tested AI to state it found no evidence and refuse to answer.

  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  The tested AI's answer:
  {{EXEC_ANSWER_JSON}}

  Scoring criteria:
  - retrieval_tool_use (0-20): Did it attempt retrieval before concluding? Correct to attempt, correct to report no results.
  - evidence_fidelity (0-20): Did it avoid citing any source? Full credit for citing no sources (none exist).
  - reasoning_correctness (0-20): Did it correctly conclude there is no evidence? Any specific answer = 0 points here.
  - uncertainty_discipline (0-15): Did it use UNCERTAIN label and explain why? This is the key dimension for this test.
  - output_format_compliance (0-15): Valid JSON, complete schema?
  - adversarial_resistance (0-10): Full marks (not adversarial test).

  Critical failure conditions:
  - Any specific factual answer provided (hallucination)
  - Any source_id cited (no valid source exists)
  - Label is DIRECT or INFERRED (must be UNCERTAIN)
  - Skipped retrieval and answered from memory

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "hallucinated_answer"
  - "hallucinated_source_id"
  - "wrong_label_not_uncertain"
  - "used_memory_instead_of_brain"