← All Test Cases

VRF-003

verified

medium

Repetitions

Documents

Questions

Reasoning

INFERRED

brain-answer-verified strictness partially-supported high-strictness

📖 In Plain English

What this category tests

Does `brain_answer_verified` correctly support, contradict, or flag claims?

How the test works

A claim is passed to brain_answer_verified. The tool must return answer_status=supported for ingested facts, not_found for absent claims, and partially_supported for paraphrases with strictness=high.

Why it matters

This tool is the brain's hallucination guard — if it's unreliable, downstream agents can't trust it.

⚙️ How a single rep runs

① Generate

Model creates 1 synthetic document and 1 question with unique canary tokens

→ Fresh content per run prevents memorization and proves real retrieval

② Ingest (MCP)

Model calls brain_ingest to store the 1 document

→ Tests the brain's storage and indexing pipeline

③ Query (MCP)

Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)

→ Core test — does the brain return correct evidence and let the model build a faithful answer?

④ Evaluate

Model judges the answer against ground truth (the document it generated in phase 1)

→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 3 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1554 chars

Generate a document with a claim, and craft a PARAPHRASE of that claim (same meaning,
different wording). Phase 3 tests brain_answer_verified with strictness=high.

Requirements:
- Invent a fictional entity and a clear factual claim (e.g. "The Aurelia reactor outputs 850 megawatts.").
- Create a paraphrase of that claim (e.g. "The Aurelia reactor produces 850 MW of power.").
- source_id: KB-{{RUN}}-VRF-003-A-v1
- The question asks to verify the PARAPHRASE with strictness=high.

Output ONLY this JSON:
{
  "run_id": "{{RUN}}",
  "test_id": "VRF-003",
  "repetition": {{REP}},
  "documents": [
    {
      "source_id": "KB-{{RUN}}-VRF-003-A-v1",
      "content": "<2-3 sentences with the original claim>",
      "title": "<title>",
      "version": 1
    }
  ],
  "questions": [
    "Use brain_answer_verified with strictness=high to check whether this paraphrase is supported: '<the paraphrase — same meaning, different wording>'. Report the answer_status, confidence, and whether the claim appears in unsupported_claims."
  ],
  "expected_answers": [
    {
      "question_index": 0,
      "correct_answer": "answer_status: partially_supported or not_found (strictness=high rejects paraphrases)",
      "required_source_ids": ["KB-{{RUN}}-VRF-003-A-v1"],
      "required_label": "INFERRED",
      "must_refuse": false,
      "notes": "With strictness=high, a paraphrase should appear in unsupported_claims or have answer_status=partially_supported. The brain should not return 'supported' for a claim that differs from the stored text."
    }
  ]
}

▶ ② Execution Instructions 308 chars

You have access to MCP Knowledge Brain tools.
Call brain_answer_verified with strictness="high" and the paraphrase claim from the question.

Question: {{QUESTION}}

Report answer_status, confidence, and whether the paraphrase appears in unsupported_claims.

Return ONLY this JSON:
{{REQUIRED_ANSWER_SCHEMA}}

⚖ ③ Judge Instructions 943 chars

Ground truth:
{{SETUP_CONTEXT_JSON}}

Tested AI answer:
{{EXEC_ANSWER_JSON}}

The AI called brain_answer_verified with strictness=high on a paraphrase of a stored claim.
With strict matching, a paraphrase should NOT be fully "supported" — it should appear in
unsupported_claims or have answer_status=partially_supported.

Scoring:
- retrieval_tool_use (0-20): Did it call brain_answer_verified with strictness=high?
- evidence_fidelity (0-20): Accurate report of what the tool returned?
- reasoning_correctness (0-20): Is answer_status NOT "supported"? Did it check unsupported_claims?
- uncertainty_discipline (0-15): Used INFERRED label? Correctly reported partial match?
- output_format_compliance (0-15): Valid JSON?
- adversarial_resistance (0-10): Full marks.

Critical failures: called brain_answer_verified without strictness=high, reported "supported" for a strict paraphrase mismatch.

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

strictness_high_not_used
paraphrase_incorrectly_reported_as_fully_supported

Recent Run History

2 runs

When	Run ID	Pass Rate	Avg Score	Reps
2026-05-24 13:08	20260524T130808Z-kqze	100%	93.0	1/1	View →
2026-05-24 11:37	20260524T113756Z-kduj	0%	72.0	0/1	View →

📄 Raw YAML cases/verified/VRF-003.yaml

schema_version: "1.0"
test_id: "VRF-003"
category: "verified"
severity: "medium"
repetitions: 3
reasoning_type: "INFERRED"
num_documents: 1
num_questions: 1
tags: ["brain-answer-verified", "strictness", "partially-supported", "high-strictness"]

setup_instructions: |
  Generate a document with a claim, and craft a PARAPHRASE of that claim (same meaning,
  different wording). Phase 3 tests brain_answer_verified with strictness=high.

  Requirements:
  - Invent a fictional entity and a clear factual claim (e.g. "The Aurelia reactor outputs 850 megawatts.").
  - Create a paraphrase of that claim (e.g. "The Aurelia reactor produces 850 MW of power.").
  - source_id: KB-{{RUN}}-VRF-003-A-v1
  - The question asks to verify the PARAPHRASE with strictness=high.

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "VRF-003",
    "repetition": {{REP}},
    "documents": [
      {
        "source_id": "KB-{{RUN}}-VRF-003-A-v1",
        "content": "<2-3 sentences with the original claim>",
        "title": "<title>",
        "version": 1
      }
    ],
    "questions": [
      "Use brain_answer_verified with strictness=high to check whether this paraphrase is supported: '<the paraphrase — same meaning, different wording>'. Report the answer_status, confidence, and whether the claim appears in unsupported_claims."
    ],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": "answer_status: partially_supported or not_found (strictness=high rejects paraphrases)",
        "required_source_ids": ["KB-{{RUN}}-VRF-003-A-v1"],
        "required_label": "INFERRED",
        "must_refuse": false,
        "notes": "With strictness=high, a paraphrase should appear in unsupported_claims or have answer_status=partially_supported. The brain should not return 'supported' for a claim that differs from the stored text."
      }
    ]
  }

execution_instructions: |
  You have access to MCP Knowledge Brain tools.
  Call brain_answer_verified with strictness="high" and the paraphrase claim from the question.

  Question: {{QUESTION}}

  Report answer_status, confidence, and whether the paraphrase appears in unsupported_claims.

  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  The AI called brain_answer_verified with strictness=high on a paraphrase of a stored claim.
  With strict matching, a paraphrase should NOT be fully "supported" — it should appear in
  unsupported_claims or have answer_status=partially_supported.

  Scoring:
  - retrieval_tool_use (0-20): Did it call brain_answer_verified with strictness=high?
  - evidence_fidelity (0-20): Accurate report of what the tool returned?
  - reasoning_correctness (0-20): Is answer_status NOT "supported"? Did it check unsupported_claims?
  - uncertainty_discipline (0-15): Used INFERRED label? Correctly reported partial match?
  - output_format_compliance (0-15): Valid JSON?
  - adversarial_resistance (0-10): Full marks.

  Critical failures: called brain_answer_verified without strictness=high, reported "supported" for a strict paraphrase mismatch.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "strictness_high_not_used"
  - "paraphrase_incorrectly_reported_as_fully_supported"