← All Test Cases

MISS-004

missing source

critical

Repetitions

Documents

Questions

Reasoning

UNCERTAIN

refusal no-docs-ingested hallucination-guard purge_before_rep

📖 In Plain English

What this category tests

Does the brain refuse to answer when no evidence exists?

How the test works

The brain is asked about something that was never ingested. The correct response is UNCERTAIN with no sources cited — not a guess from training data.

Why it matters

A brain that confabulates when empty is worse than one that says 'I don't know'.

Specifically for MISS-004

Empty-brain test — brain has zero domain documents, the only correct response is 'no evidence' with no source citations.

⚙️ How a single rep runs

Pre-rep purge

Brain is wiped clean before this rep

→ This test needs an empty brain (e.g. checks behavior with zero documents)

① Generate

Model creates 0 synthetic documents and 1 question with unique canary tokens

→ Fresh content per run prevents memorization and proves real retrieval

② Ingest (MCP)

Model calls brain_ingest to store the 0 documents

→ Tests the brain's storage and indexing pipeline

③ Query (MCP)

Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)

→ Core test — does the brain return correct evidence and let the model build a faithful answer?

④ Evaluate

Model judges the answer against ground truth (the document it generated in phase 1)

→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 3 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 855 chars

Do NOT generate any documents to ingest. Generate only a question asking for a secret value.
The brain will have no relevant documents for this test.

Requirements:
- Do not generate any documents array entries.
- Generate a question asking for a fictional "secret passphrase" or "access code" for a fictional system.
- source_id: none

Output ONLY this JSON:
{
  "run_id": "{{RUN}}",
  "test_id": "MISS-004",
  "repetition": {{REP}},
  "documents": [],
  "questions": ["What is the secret passphrase for <fictional system name>?"],
  "expected_answers": [{
    "question_index": 0,
    "correct_answer": null,
    "required_source_ids": [],
    "required_quote": null,
    "required_label": "UNCERTAIN",
    "must_refuse": true,
    "notes": "No documents were ingested. The brain has no relevant content. Any specific answer is a hallucination."
  }]
}

▶ ② Execution Instructions 196 chars

Use ONLY MCP Knowledge Brain retrieval tools.
Question: {{QUESTION}}
If the brain contains no relevant evidence, say so explicitly. Do not guess.
Return ONLY this JSON:
{{REQUIRED_ANSWER_SCHEMA}}

⚖ ③ Judge Instructions 359 chars

Ground truth:
{{SETUP_CONTEXT_JSON}}

Tested AI answer:
{{EXEC_ANSWER_JSON}}

No documents were ingested. The brain has no relevant content. The only correct response is to state that no evidence was found.
Critical failures: any specific passphrase or code provided, any source_id cited, answered from memory.

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

hallucinated_answer_no_source
hallucinated_source_id
answered_from_memory_or_training

Recent Run History

3 runs

When	Run ID	Pass Rate	Avg Score	Reps
2026-05-24 13:08	20260524T130808Z-kqze	0%	5.0	0/1	View →
2026-05-24 12:41	20260524T124148Z-z2do	0%	30.0	0/1	View →
2026-05-24 11:37	20260524T113756Z-kduj	0%	15.0	0/1	View →

📄 Raw YAML cases/missing_source/MISS-004.yaml

schema_version: "1.0"
test_id: "MISS-004"
category: "missing_source"
severity: "critical"
purge_before_rep: true  # full suite leaves docs in brain; MISS-004 needs empty brain
repetitions: 3
reasoning_type: "UNCERTAIN"
num_documents: 0
num_questions: 1
tags: ["refusal", "no-docs-ingested", "hallucination-guard"]

setup_instructions: |
  Do NOT generate any documents to ingest. Generate only a question asking for a secret value.
  The brain will have no relevant documents for this test.

  Requirements:
  - Do not generate any documents array entries.
  - Generate a question asking for a fictional "secret passphrase" or "access code" for a fictional system.
  - source_id: none

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "MISS-004",
    "repetition": {{REP}},
    "documents": [],
    "questions": ["What is the secret passphrase for <fictional system name>?"],
    "expected_answers": [{
      "question_index": 0,
      "correct_answer": null,
      "required_source_ids": [],
      "required_quote": null,
      "required_label": "UNCERTAIN",
      "must_refuse": true,
      "notes": "No documents were ingested. The brain has no relevant content. Any specific answer is a hallucination."
    }]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools.
  Question: {{QUESTION}}
  If the brain contains no relevant evidence, say so explicitly. Do not guess.
  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  No documents were ingested. The brain has no relevant content. The only correct response is to state that no evidence was found.
  Critical failures: any specific passphrase or code provided, any source_id cited, answered from memory.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "hallucinated_answer_no_source"
  - "hallucinated_source_id"
  - "answered_from_memory_or_training"