← All Test Cases

CON-003

conflicting evidence

high

Repetitions

Documents

Questions

Reasoning

UNCERTAIN

multi-source-conflict partial-agreement uncertainty

📖 In Plain English

What this category tests

When two documents disagree, does the brain surface the contradiction?

How the test works

Multiple documents are ingested with conflicting facts about the same entity. The test checks the brain reports the conflict rather than silently picking one value.

Why it matters

Silent picking causes hidden errors. Users must know when sources disagree.

Specifically for CON-003

Three-document conflict — all 3 must be cited; missing any one is a critical failure.

⚙️ How a single rep runs

① Generate

Model creates 3 synthetic documents and 1 question with unique canary tokens

→ Fresh content per run prevents memorization and proves real retrieval

② Ingest (MCP)

Model calls brain_ingest to store the 3 documents

→ Tests the brain's storage and indexing pipeline

③ Query (MCP)

Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)

→ Core test — does the brain return correct evidence and let the model build a faithful answer?

④ Evaluate

Model judges the answer against ground truth (the document it generated in phase 1)

→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 3 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1355 chars

Generate three documents where two agree and one disagrees. The brain must note the disagreement.

Requirements:
- Doc A: "<Entity> value is <X>."
- Doc B: "<Entity> value is <X>." (agrees with A)
- Doc C: "<Entity> value is <Y>." (disagrees — different value)
- source_ids: KB-{{RUN}}-CON-003-A-v1, KB-{{RUN}}-CON-003-B-v1, KB-{{RUN}}-CON-003-C-v1.
- Question: "What is the <Entity> value?"

Output ONLY this JSON:
{
  "run_id": "{{RUN}}",
  "test_id": "CON-003",
  "repetition": {{REP}},
  "documents": [
    {"source_id": "KB-{{RUN}}-CON-003-A-v1", "content": "<Entity> value is <X>.", "title": "<title>", "version": 1},
    {"source_id": "KB-{{RUN}}-CON-003-B-v1", "content": "<Entity> value is <X>.", "title": "<title>", "version": 1},
    {"source_id": "KB-{{RUN}}-CON-003-C-v1", "content": "<Entity> value is <Y>.", "title": "<title>", "version": 1}
  ],
  "questions": ["What is the <Entity> value?"],
  "expected_answers": [{
    "question_index": 0,
    "correct_answer": null,
    "required_source_ids": ["KB-{{RUN}}-CON-003-A-v1", "KB-{{RUN}}-CON-003-B-v1", "KB-{{RUN}}-CON-003-C-v1"],
    "required_quote": null,
    "required_label": "UNCERTAIN",
    "must_refuse": false,
    "notes": "2 sources say X, 1 says Y. The conflict must be surfaced even though a majority agree. The brain must not silently pick X just because 2/3 agree."
  }]
}

▶ ② Execution Instructions 312 chars

Use ONLY MCP Knowledge Brain retrieval tools with domain="brain-test".


CRITICAL: This test has 3 documents (A, B, and C). Your answer MUST cite all 3 source IDs. If you can only find 2, explicitly search for the third before answering.

Question: {{QUESTION}}
Return ONLY this JSON:
{{REQUIRED_ANSWER_SCHEMA}}

⚖ ③ Judge Instructions 349 chars

Ground truth:
{{SETUP_CONTEXT_JSON}}

Tested AI answer:
{{EXEC_ANSWER_JSON}}

2 docs agree (X), 1 disagrees (Y). The conflict must be reported even though majority agree. UNCERTAIN required.
Critical failures: silently chose X without mentioning Y, cited fewer than 3 sources, hallucinated source_id.

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

silently_chose_majority_without_reporting_conflict
cited_fewer_than_3_sources
hallucinated_source_id
wrong_label_not_uncertain

Recent Run History

3 runs

When	Run ID	Pass Rate	Avg Score	Reps
2026-05-24 13:08	20260524T130808Z-kqze	0%	20.0	0/1	View →
2026-05-24 12:41	20260524T124148Z-z2do	100%	100.0	1/1	View →
2026-05-24 11:37	20260524T113756Z-kduj	100%	100.0	1/1	View →

📄 Raw YAML cases/conflicting_evidence/CON-003.yaml

schema_version: "1.0"
test_id: "CON-003"
category: "conflicting_evidence"
severity: "high"
repetitions: 3
reasoning_type: "UNCERTAIN"
num_documents: 3
num_questions: 1
tags: ["multi-source-conflict", "partial-agreement", "uncertainty"]

setup_instructions: |
  Generate three documents where two agree and one disagrees. The brain must note the disagreement.

  Requirements:
  - Doc A: "<Entity> value is <X>."
  - Doc B: "<Entity> value is <X>." (agrees with A)
  - Doc C: "<Entity> value is <Y>." (disagrees — different value)
  - source_ids: KB-{{RUN}}-CON-003-A-v1, KB-{{RUN}}-CON-003-B-v1, KB-{{RUN}}-CON-003-C-v1.
  - Question: "What is the <Entity> value?"

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "CON-003",
    "repetition": {{REP}},
    "documents": [
      {"source_id": "KB-{{RUN}}-CON-003-A-v1", "content": "<Entity> value is <X>.", "title": "<title>", "version": 1},
      {"source_id": "KB-{{RUN}}-CON-003-B-v1", "content": "<Entity> value is <X>.", "title": "<title>", "version": 1},
      {"source_id": "KB-{{RUN}}-CON-003-C-v1", "content": "<Entity> value is <Y>.", "title": "<title>", "version": 1}
    ],
    "questions": ["What is the <Entity> value?"],
    "expected_answers": [{
      "question_index": 0,
      "correct_answer": null,
      "required_source_ids": ["KB-{{RUN}}-CON-003-A-v1", "KB-{{RUN}}-CON-003-B-v1", "KB-{{RUN}}-CON-003-C-v1"],
      "required_quote": null,
      "required_label": "UNCERTAIN",
      "must_refuse": false,
      "notes": "2 sources say X, 1 says Y. The conflict must be surfaced even though a majority agree. The brain must not silently pick X just because 2/3 agree."
    }]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools with domain="brain-test".
  

  CRITICAL: This test has 3 documents (A, B, and C). Your answer MUST cite all 3 source IDs. If you can only find 2, explicitly search for the third before answering.

  Question: {{QUESTION}}
  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  2 docs agree (X), 1 disagrees (Y). The conflict must be reported even though majority agree. UNCERTAIN required.
  Critical failures: silently chose X without mentioning Y, cited fewer than 3 sources, hallucinated source_id.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "silently_chose_majority_without_reporting_conflict"
  - "cited_fewer_than_3_sources"
  - "hallucinated_source_id"
  - "wrong_label_not_uncertain"