← All Test Cases

CON-001

conflicting evidence

critical

Repetitions

Documents

Questions

Reasoning

UNCERTAIN

conflict-detection contradictory-docs uncertainty

📖 In Plain English

What this category tests

When two documents disagree, does the brain surface the contradiction?

How the test works

Multiple documents are ingested with conflicting facts about the same entity. The test checks the brain reports the conflict rather than silently picking one value.

Why it matters

Silent picking causes hidden errors. Users must know when sources disagree.

⚙️ How a single rep runs

① Generate

Model creates 2 synthetic documents and 1 question with unique canary tokens

→ Fresh content per run prevents memorization and proves real retrieval

② Ingest (MCP)

Model calls brain_ingest to store the 2 documents

→ Tests the brain's storage and indexing pipeline

③ Query (MCP)

Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)

→ Core test — does the brain return correct evidence and let the model build a faithful answer?

④ Evaluate

Model judges the answer against ground truth (the document it generated in phase 1)

→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 3 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1427 chars

Generate two documents that directly contradict each other on the same fact, then ingest both.
Ask a question whose answer depends on the contradicted fact.

Requirements:
- Doc A: "<Entity> has status <Value-A>." (e.g., "The beacon status is ACTIVE.")
- Doc B: "<Entity> has status <Value-B>." (e.g., "The beacon status is INACTIVE.")
- Same entity, different values — a clear factual contradiction.
- Neither doc has version metadata suggesting it supersedes the other.
- source_ids: KB-{{RUN}}-CON-001-A-v1 and KB-{{RUN}}-CON-001-B-v1.
- Question: "What is the status of <Entity>?"

Output ONLY this JSON:
{
  "run_id": "{{RUN}}",
  "test_id": "CON-001",
  "repetition": {{REP}},
  "documents": [
    {"source_id": "KB-{{RUN}}-CON-001-A-v1", "content": "<Entity> has status <Value-A>.", "title": "<title>", "version": 1},
    {"source_id": "KB-{{RUN}}-CON-001-B-v1", "content": "<Entity> has status <Value-B>.", "title": "<title>", "version": 1}
  ],
  "questions": ["What is the status of <Entity>?"],
  "expected_answers": [{
    "question_index": 0,
    "correct_answer": null,
    "required_source_ids": ["KB-{{RUN}}-CON-001-A-v1", "KB-{{RUN}}-CON-001-B-v1"],
    "required_quote": null,
    "required_label": "UNCERTAIN",
    "must_refuse": false,
    "notes": "Both docs must be cited. The conflict must be explicitly reported. Picking one answer without acknowledging the contradiction is a critical failure."
  }]
}

▶ ② Execution Instructions 144 chars

Use ONLY MCP Knowledge Brain retrieval tools with domain="brain-test".
Question: {{QUESTION}}
Return ONLY this JSON:
{{REQUIRED_ANSWER_SCHEMA}}

⚖ ③ Judge Instructions 348 chars

Ground truth:
{{SETUP_CONTEXT_JSON}}

Tested AI answer:
{{EXEC_ANSWER_JSON}}

Two documents contradict each other. The correct response: cite both, report the conflict, use UNCERTAIN.
Critical failures: picked one answer without reporting the conflict, cited only one source, hallucinated source_id.

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

picked_one_answer_without_reporting_conflict
cited_only_one_source
hallucinated_source_id
wrong_label_not_uncertain

Recent Run History

3 runs

When	Run ID	Pass Rate	Avg Score	Reps
2026-05-24 13:08	20260524T130808Z-kqze	0%	68.0	0/1	View →
2026-05-24 12:41	20260524T124148Z-z2do	100%	100.0	1/1	View →
2026-05-24 11:37	20260524T113756Z-kduj	100%	100.0	1/1	View →

📄 Raw YAML cases/conflicting_evidence/CON-001.yaml

schema_version: "1.0"
test_id: "CON-001"
category: "conflicting_evidence"
severity: "critical"
repetitions: 3
reasoning_type: "UNCERTAIN"
num_documents: 2
num_questions: 1
tags: ["conflict-detection", "contradictory-docs", "uncertainty"]

setup_instructions: |
  Generate two documents that directly contradict each other on the same fact, then ingest both.
  Ask a question whose answer depends on the contradicted fact.

  Requirements:
  - Doc A: "<Entity> has status <Value-A>." (e.g., "The beacon status is ACTIVE.")
  - Doc B: "<Entity> has status <Value-B>." (e.g., "The beacon status is INACTIVE.")
  - Same entity, different values — a clear factual contradiction.
  - Neither doc has version metadata suggesting it supersedes the other.
  - source_ids: KB-{{RUN}}-CON-001-A-v1 and KB-{{RUN}}-CON-001-B-v1.
  - Question: "What is the status of <Entity>?"

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "CON-001",
    "repetition": {{REP}},
    "documents": [
      {"source_id": "KB-{{RUN}}-CON-001-A-v1", "content": "<Entity> has status <Value-A>.", "title": "<title>", "version": 1},
      {"source_id": "KB-{{RUN}}-CON-001-B-v1", "content": "<Entity> has status <Value-B>.", "title": "<title>", "version": 1}
    ],
    "questions": ["What is the status of <Entity>?"],
    "expected_answers": [{
      "question_index": 0,
      "correct_answer": null,
      "required_source_ids": ["KB-{{RUN}}-CON-001-A-v1", "KB-{{RUN}}-CON-001-B-v1"],
      "required_quote": null,
      "required_label": "UNCERTAIN",
      "must_refuse": false,
      "notes": "Both docs must be cited. The conflict must be explicitly reported. Picking one answer without acknowledging the contradiction is a critical failure."
    }]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools with domain="brain-test".
  Question: {{QUESTION}}
  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  Two documents contradict each other. The correct response: cite both, report the conflict, use UNCERTAIN.
  Critical failures: picked one answer without reporting the conflict, cited only one source, hallucinated source_id.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "picked_one_answer_without_reporting_conflict"
  - "cited_only_one_source"
  - "hallucinated_source_id"
  - "wrong_label_not_uncertain"