← All Test Cases

CON-002

conflicting evidence

high

Repetitions

Documents

Questions

Reasoning

DIRECT

conflict-resolution versioned-contradiction supersession

📖 In Plain English

What this category tests

When two documents disagree, does the brain surface the contradiction?

How the test works

Multiple documents are ingested with conflicting facts about the same entity. The test checks the brain reports the conflict rather than silently picking one value.

Why it matters

Silent picking causes hidden errors. Users must know when sources disagree.

⚙️ How a single rep runs

① Generate

Model creates 2 synthetic documents and 1 question with unique canary tokens

→ Fresh content per run prevents memorization and proves real retrieval

② Ingest (MCP)

Model calls brain_ingest to store the 2 documents

→ Tests the brain's storage and indexing pipeline

③ Query (MCP)

Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)

→ Core test — does the brain return correct evidence and let the model build a faithful answer?

④ Evaluate

Model judges the answer against ground truth (the document it generated in phase 1)

→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 3 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1317 chars

Generate two contradictory documents where one explicitly supersedes the other (via version metadata).
The brain should recognize the supersession and answer with the newer version.

Requirements:
- Doc A (v1): "<Entity> limit is <Value-A>." — older version
- Doc B (v2): "<Entity> limit is <Value-B>. This supersedes version 1." — explicitly newer
- source_ids: KB-{{RUN}}-CON-002-A-v1 and KB-{{RUN}}-CON-002-B-v2 (note v2).
- Question: "What is the current <Entity> limit?"

Output ONLY this JSON:
{
  "run_id": "{{RUN}}",
  "test_id": "CON-002",
  "repetition": {{REP}},
  "documents": [
    {"source_id": "KB-{{RUN}}-CON-002-A-v1", "content": "<Entity> limit is <Value-A>.", "title": "<title>", "version": 1},
    {"source_id": "KB-{{RUN}}-CON-002-B-v2", "content": "<Entity> limit is <Value-B>. This supersedes version 1.", "title": "<title>", "version": 2}
  ],
  "questions": ["What is the current <Entity> limit?"],
  "expected_answers": [{
    "question_index": 0,
    "correct_answer": "<Value-B>",
    "required_source_ids": ["KB-{{RUN}}-CON-002-B-v2"],
    "required_quote": "<exact sentence from Doc B>",
    "required_label": "DIRECT",
    "must_refuse": false,
    "notes": "Doc B explicitly supersedes Doc A. Answer must be Value-B from Doc B. Answering Value-A (stale) is a critical failure."
  }]
}

▶ ② Execution Instructions 144 chars

Use ONLY MCP Knowledge Brain retrieval tools with domain="brain-test".
Question: {{QUESTION}}
Return ONLY this JSON:
{{REQUIRED_ANSWER_SCHEMA}}

⚖ ③ Judge Instructions 306 chars

Ground truth:
{{SETUP_CONTEXT_JSON}}

Tested AI answer:
{{EXEC_ANSWER_JSON}}

Doc B explicitly supersedes Doc A. The current answer is Value-B from Doc B.
Critical failures: answered with stale Value-A, did not note the supersession, hallucinated source_id.

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

answered_with_stale_value
hallucinated_source_id
ignored_supersession_metadata

Recent Run History

3 runs

When	Run ID	Pass Rate	Avg Score	Reps
2026-05-24 13:08	20260524T130808Z-kqze	100%	100.0	1/1	View →
2026-05-24 12:41	20260524T124148Z-z2do	100%	100.0	1/1	View →
2026-05-24 11:37	20260524T113756Z-kduj	100%	100.0	1/1	View →

📄 Raw YAML cases/conflicting_evidence/CON-002.yaml

schema_version: "1.0"
test_id: "CON-002"
category: "conflicting_evidence"
severity: "high"
repetitions: 3
reasoning_type: "DIRECT"
num_documents: 2
num_questions: 1
tags: ["conflict-resolution", "versioned-contradiction", "supersession"]

setup_instructions: |
  Generate two contradictory documents where one explicitly supersedes the other (via version metadata).
  The brain should recognize the supersession and answer with the newer version.

  Requirements:
  - Doc A (v1): "<Entity> limit is <Value-A>." — older version
  - Doc B (v2): "<Entity> limit is <Value-B>. This supersedes version 1." — explicitly newer
  - source_ids: KB-{{RUN}}-CON-002-A-v1 and KB-{{RUN}}-CON-002-B-v2 (note v2).
  - Question: "What is the current <Entity> limit?"

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "CON-002",
    "repetition": {{REP}},
    "documents": [
      {"source_id": "KB-{{RUN}}-CON-002-A-v1", "content": "<Entity> limit is <Value-A>.", "title": "<title>", "version": 1},
      {"source_id": "KB-{{RUN}}-CON-002-B-v2", "content": "<Entity> limit is <Value-B>. This supersedes version 1.", "title": "<title>", "version": 2}
    ],
    "questions": ["What is the current <Entity> limit?"],
    "expected_answers": [{
      "question_index": 0,
      "correct_answer": "<Value-B>",
      "required_source_ids": ["KB-{{RUN}}-CON-002-B-v2"],
      "required_quote": "<exact sentence from Doc B>",
      "required_label": "DIRECT",
      "must_refuse": false,
      "notes": "Doc B explicitly supersedes Doc A. Answer must be Value-B from Doc B. Answering Value-A (stale) is a critical failure."
    }]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools with domain="brain-test".
  Question: {{QUESTION}}
  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  Doc B explicitly supersedes Doc A. The current answer is Value-B from Doc B.
  Critical failures: answered with stale Value-A, did not note the supersession, hallucinated source_id.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "answered_with_stale_value"
  - "hallucinated_source_id"
  - "ignored_supersession_metadata"