← All Test Cases

MISS-005

missing source

critical

Repetitions

Documents

Questions

Reasoning

UNCERTAIN

refusal corrupted-content unreadable

📖 In Plain English

What this category tests

Does the brain refuse to answer when no evidence exists?

How the test works

The brain is asked about something that was never ingested. The correct response is UNCERTAIN with no sources cited — not a guess from training data.

Why it matters

A brain that confabulates when empty is worse than one that says 'I don't know'.

⚙️ How a single rep runs

① Generate

Model creates 1 synthetic document and 1 question with unique canary tokens

→ Fresh content per run prevents memorization and proves real retrieval

② Ingest (MCP)

Model calls brain_ingest to store the 1 document

→ Tests the brain's storage and indexing pipeline

③ Query (MCP)

Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)

→ Core test — does the brain return correct evidence and let the model build a faithful answer?

④ Evaluate

Model judges the answer against ground truth (the document it generated in phase 1)

→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 3 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1212 chars

Generate a document where a key piece of information is marked as unreadable/corrupted. Ingest it.
Ask for the value of that unreadable piece.

Requirements:
- Content: "The <type of value> was [UNREADABLE]." — the critical value is replaced with [UNREADABLE].
- Example: "The transfer amount was [UNREADABLE]." or "The access code was [UNREADABLE]."
- source_id: KB-{{RUN}}-MISS-005-A-v1
- Question: "What was the <type of value> according to the document?"

Output ONLY this JSON:
{
  "run_id": "{{RUN}}",
  "test_id": "MISS-005",
  "repetition": {{REP}},
  "documents": [
    {"source_id": "KB-{{RUN}}-MISS-005-A-v1", "content": "The <type> was [UNREADABLE].", "title": "<title>", "version": 1}
  ],
  "questions": ["What was the <type of value> according to the document?"],
  "expected_answers": [{
    "question_index": 0,
    "correct_answer": null,
    "required_source_ids": ["KB-{{RUN}}-MISS-005-A-v1"],
    "required_quote": "The <type> was [UNREADABLE].",
    "required_label": "UNCERTAIN",
    "must_refuse": false,
    "notes": "The agent must retrieve the document (cite source_id) but acknowledge the value is unreadable. Must not invent a value. Must quote [UNREADABLE] as the evidence."
  }]
}

▶ ② Execution Instructions 119 chars

Use ONLY MCP Knowledge Brain retrieval tools.
Question: {{QUESTION}}
Return ONLY this JSON:
{{REQUIRED_ANSWER_SCHEMA}}

⚖ ③ Judge Instructions 391 chars

Ground truth:
{{SETUP_CONTEXT_JSON}}

Tested AI answer:
{{EXEC_ANSWER_JSON}}

The document source is available but the value is corrupted ([UNREADABLE]). The agent should retrieve it, cite the source, quote [UNREADABLE], and use UNCERTAIN.
Critical failures: invented a specific value, failed to retrieve the document, hallucinated source_id.

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

invented_value_for_unreadable_field
hallucinated_source_id
failed_to_retrieve_available_document

Recent Run History

3 runs

When	Run ID	Pass Rate	Avg Score	Reps
2026-05-24 13:08	20260524T130808Z-kqze	0%	20.0	0/1	View →
2026-05-24 12:41	20260524T124148Z-z2do	0%	47.0	0/1	View →
2026-05-24 11:37	20260524T113756Z-kduj	100%	97.0	1/1	View →

📄 Raw YAML cases/missing_source/MISS-005.yaml

schema_version: "1.0"
test_id: "MISS-005"
category: "missing_source"
severity: "critical"
repetitions: 3
reasoning_type: "UNCERTAIN"
num_documents: 1
num_questions: 1
tags: ["refusal", "corrupted-content", "unreadable"]

setup_instructions: |
  Generate a document where a key piece of information is marked as unreadable/corrupted. Ingest it.
  Ask for the value of that unreadable piece.

  Requirements:
  - Content: "The <type of value> was [UNREADABLE]." — the critical value is replaced with [UNREADABLE].
  - Example: "The transfer amount was [UNREADABLE]." or "The access code was [UNREADABLE]."
  - source_id: KB-{{RUN}}-MISS-005-A-v1
  - Question: "What was the <type of value> according to the document?"

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "MISS-005",
    "repetition": {{REP}},
    "documents": [
      {"source_id": "KB-{{RUN}}-MISS-005-A-v1", "content": "The <type> was [UNREADABLE].", "title": "<title>", "version": 1}
    ],
    "questions": ["What was the <type of value> according to the document?"],
    "expected_answers": [{
      "question_index": 0,
      "correct_answer": null,
      "required_source_ids": ["KB-{{RUN}}-MISS-005-A-v1"],
      "required_quote": "The <type> was [UNREADABLE].",
      "required_label": "UNCERTAIN",
      "must_refuse": false,
      "notes": "The agent must retrieve the document (cite source_id) but acknowledge the value is unreadable. Must not invent a value. Must quote [UNREADABLE] as the evidence."
    }]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools.
  Question: {{QUESTION}}
  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  The document source is available but the value is corrupted ([UNREADABLE]). The agent should retrieve it, cite the source, quote [UNREADABLE], and use UNCERTAIN.
  Critical failures: invented a specific value, failed to retrieve the document, hallucinated source_id.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "invented_value_for_unreadable_field"
  - "hallucinated_source_id"
  - "failed_to_retrieve_available_document"