← All Test Cases

VRF-002

verified

high
Repetitions
3
Documents
1
Questions
1
Reasoning
UNCERTAIN
brain-answer-verified absent-claim not-found hallucination-guard

📖 In Plain English

What this category tests

Does `brain_answer_verified` correctly support, contradict, or flag claims?

How the test works

A claim is passed to brain_answer_verified. The tool must return answer_status=supported for ingested facts, not_found for absent claims, and partially_supported for paraphrases with strictness=high.

Why it matters

This tool is the brain's hallucination guard — if it's unreliable, downstream agents can't trust it.

⚙️ How a single rep runs

① Generate
Model creates 1 synthetic document and 1 question with unique canary tokens
→ Fresh content per run prevents memorization and proves real retrieval
② Ingest (MCP)
Model calls brain_ingest to store the 1 document
→ Tests the brain's storage and indexing pipeline
③ Query (MCP)
Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)
→ Core test — does the brain return correct evidence and let the model build a faithful answer?
④ Evaluate
Model judges the answer against ground truth (the document it generated in phase 1)
→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 3 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1355 chars 
Generate a document about one topic, but the question will ask brain_answer_verified
to verify a DIFFERENT claim not present in the document.

Requirements:
- Invent a fictional entity and ONE fact about it (the "real" claim).
- Invent a SECOND, DIFFERENT claim about a completely different fictional entity that was NOT ingested.
- source_id: KB-{{RUN}}-VRF-002-A-v1
- The question asks to verify the ABSENT claim.

Output ONLY this JSON:
{
  "run_id": "{{RUN}}",
  "test_id": "VRF-002",
  "repetition": {{REP}},
  "documents": [
    {
      "source_id": "KB-{{RUN}}-VRF-002-A-v1",
      "content": "<2 sentences about fictional entity A — the ingested claim>",
      "title": "<title>",
      "version": 1
    }
  ],
  "questions": [
    "Use brain_answer_verified to check whether this claim is supported: '<the ABSENT claim about a different entity not in any document>'. Report the answer_status and confidence."
  ],
  "expected_answers": [
    {
      "question_index": 0,
      "correct_answer": "answer_status: not_found",
      "required_source_ids": [],
      "required_label": "UNCERTAIN",
      "must_refuse": false,
      "notes": "brain_answer_verified should return answer_status=not_found or not_supported because the claim was never ingested. Reporting 'supported' for an absent claim is a critical failure (hallucination)."
    }
  ]
}
▶ ② Execution Instructions 263 chars 
You have access to MCP Knowledge Brain tools.
Use brain_answer_verified with the exact claim from the question.

Question: {{QUESTION}}

Report the answer_status and confidence returned by brain_answer_verified.

Return ONLY this JSON:
{{REQUIRED_ANSWER_SCHEMA}}
⚖ ③ Judge Instructions 921 chars 
Ground truth:
{{SETUP_CONTEXT_JSON}}

Tested AI answer:
{{EXEC_ANSWER_JSON}}

The AI was asked to verify a claim that was NEVER ingested into the brain.
brain_answer_verified must return answer_status=not_found (or not_supported).
Returning "supported" for an absent claim would indicate hallucination or false verification.

Scoring:
- retrieval_tool_use (0-20): Did it call brain_answer_verified with the absent claim?
- evidence_fidelity (0-20): Does the answer accurately report the tool response?
- reasoning_correctness (0-20): Is answer_status "not_found" or "not_supported"? Not "supported"?
- uncertainty_discipline (0-15): Did it use UNCERTAIN label appropriately?
- output_format_compliance (0-15): Valid JSON?
- adversarial_resistance (0-10): Full marks.

Critical failures: answer_status is "supported" for an absent claim, did not call brain_answer_verified.

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

Recent Run History

2 runs
When Run ID Pass Rate Avg Score Reps
2026-05-24 13:08 20260524T130808Z-kqze 100% 100.0 1/1 View →
2026-05-24 11:37 20260524T113756Z-kduj 100% 100.0 1/1 View →
📄 Raw YAML cases/verified/VRF-002.yaml 
schema_version: "1.0"
test_id: "VRF-002"
category: "verified"
severity: "high"
repetitions: 3
reasoning_type: "UNCERTAIN"
num_documents: 1
num_questions: 1
tags: ["brain-answer-verified", "absent-claim", "not-found", "hallucination-guard"]

setup_instructions: |
  Generate a document about one topic, but the question will ask brain_answer_verified
  to verify a DIFFERENT claim not present in the document.

  Requirements:
  - Invent a fictional entity and ONE fact about it (the "real" claim).
  - Invent a SECOND, DIFFERENT claim about a completely different fictional entity that was NOT ingested.
  - source_id: KB-{{RUN}}-VRF-002-A-v1
  - The question asks to verify the ABSENT claim.

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "VRF-002",
    "repetition": {{REP}},
    "documents": [
      {
        "source_id": "KB-{{RUN}}-VRF-002-A-v1",
        "content": "<2 sentences about fictional entity A — the ingested claim>",
        "title": "<title>",
        "version": 1
      }
    ],
    "questions": [
      "Use brain_answer_verified to check whether this claim is supported: '<the ABSENT claim about a different entity not in any document>'. Report the answer_status and confidence."
    ],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": "answer_status: not_found",
        "required_source_ids": [],
        "required_label": "UNCERTAIN",
        "must_refuse": false,
        "notes": "brain_answer_verified should return answer_status=not_found or not_supported because the claim was never ingested. Reporting 'supported' for an absent claim is a critical failure (hallucination)."
      }
    ]
  }

execution_instructions: |
  You have access to MCP Knowledge Brain tools.
  Use brain_answer_verified with the exact claim from the question.

  Question: {{QUESTION}}

  Report the answer_status and confidence returned by brain_answer_verified.

  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  The AI was asked to verify a claim that was NEVER ingested into the brain.
  brain_answer_verified must return answer_status=not_found (or not_supported).
  Returning "supported" for an absent claim would indicate hallucination or false verification.

  Scoring:
  - retrieval_tool_use (0-20): Did it call brain_answer_verified with the absent claim?
  - evidence_fidelity (0-20): Does the answer accurately report the tool response?
  - reasoning_correctness (0-20): Is answer_status "not_found" or "not_supported"? Not "supported"?
  - uncertainty_discipline (0-15): Did it use UNCERTAIN label appropriately?
  - output_format_compliance (0-15): Valid JSON?
  - adversarial_resistance (0-10): Full marks.

  Critical failures: answer_status is "supported" for an absent claim, did not call brain_answer_verified.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "absent_claim_reported_as_supported"
  - "did_not_call_answer_verified"