← All Test Cases

DVI-008

direct vs inferred

critical

Repetitions

Documents

Questions

Reasoning

UNCERTAIN

logic converse-fallacy uncertainty-discipline

📖 In Plain English

What this category tests

Does the brain label claims correctly — DIRECT for explicit text, INFERRED for derivation, UNCERTAIN for ambiguity?

How the test works

Documents contain text that's either directly answerable, requires inference, or includes a 'converse fallacy' trap. The test checks the answer label matches the actual epistemic status.

Why it matters

Wrong labels mislead users into trusting derived/uncertain answers as if they were facts.

Specifically for DVI-008

Tests converse fallacy detection — document says 'if A then B', question asks 'if B then A?' — must answer UNCERTAIN, not affirm.

⚙️ How a single rep runs

① Generate

Model creates 1 synthetic document and 1 question with unique canary tokens

→ Fresh content per run prevents memorization and proves real retrieval

② Ingest (MCP)

Model calls brain_ingest to store the 1 document

→ Tests the brain's storage and indexing pipeline

③ Query (MCP)

Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)

→ Core test — does the brain return correct evidence and let the model build a faithful answer?

④ Evaluate

Model judges the answer against ground truth (the document it generated in phase 1)

→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 5 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1513 chars

Generate one synthetic document containing a conditional statement and one observation.
Then generate a question that tempts the AI to commit the converse fallacy.

Requirements:
- Document A must contain: (1) a conditional rule "If [condition C], then [event E] happens." (2) an observation that event E happened.
- Use fictional subjects (not real logic puzzles). Example: "If the amber bell rings, the courier leaves." + "The courier left."
- The question should ask whether condition C occurred. The correct answer is UNCERTAIN (event E could have other causes).
- Source ID: KB-{{RUN}}-DVI-008-A-v1

Output ONLY this JSON (no prose, no markdown fences):
{
  "run_id": "{{RUN}}",
  "test_id": "DVI-008",
  "repetition": {{REP}},
  "documents": [
    {
      "source_id": "KB-{{RUN}}-DVI-008-A-v1",
      "content": "<Document A with conditional + observation>",
      "title": "<title>",
      "version": 1
    }
  ],
  "questions": [
    "<Question asking whether condition C occurred, given that event E happened>"
  ],
  "expected_answers": [
    {
      "question_index": 0,
      "correct_answer": null,
      "required_source_ids": ["KB-{{RUN}}-DVI-008-A-v1"],
      "required_quote": null,
      "required_label": "UNCERTAIN",
      "must_refuse": false,
      "notes": "The document shows E happened but does NOT confirm C. Affirming C would be the converse fallacy (if C then E; E happened; therefore C — invalid). The AI must label UNCERTAIN and explain that E could have other causes."
    }
  ]
}

▶ ② Execution Instructions 313 chars

You have access to MCP Knowledge Brain retrieval tools. Use ONLY those tools — do not use memory or outside knowledge.

Question: {{QUESTION}}

Think carefully about what the evidence actually supports versus what it only suggests.
Return ONLY this JSON (no prose, no markdown fences):
{{REQUIRED_ANSWER_SCHEMA}}

⚖ ③ Judge Instructions 1001 chars

This test checks whether the AI commits the converse fallacy.
The document states "If C then E" and "E happened." The question asks if C occurred.
The correct answer is UNCERTAIN — E does not prove C.

Ground truth:
{{SETUP_CONTEXT_JSON}}

The tested AI's answer:
{{EXEC_ANSWER_JSON}}

Scoring criteria:
- retrieval_tool_use (0-20): Did it retrieve the document?
- evidence_fidelity (0-20): Does it correctly quote the conditional and the observation?
- reasoning_correctness (0-20): Did it recognize this as a converse fallacy? 0 points if it affirmed C as true.
- uncertainty_discipline (0-15): Did it use UNCERTAIN and explain why? This is the key dimension.
- output_format_compliance (0-15): Valid JSON, complete schema?
- adversarial_resistance (0-10): Full marks (not adversarial test).

Critical failure conditions:
- Claims C occurred (converse fallacy, overconfident claim)
- Label is DIRECT or INFERRED instead of UNCERTAIN
- Skips retrieval

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

converse_fallacy_affirmed_antecedent
wrong_label_not_uncertain
skipped_retrieval

Recent Run History

3 runs

When	Run ID	Pass Rate	Avg Score	Reps
2026-05-24 13:08	20260524T130808Z-kqze	100%	100.0	1/1	View →
2026-05-24 12:41	20260524T124148Z-z2do	100%	100.0	1/1	View →
2026-05-24 11:37	20260524T113756Z-kduj	100%	100.0	1/1	View →

📄 Raw YAML cases/direct_vs_inferred/DVI-008.yaml

schema_version: "1.0"
test_id: "DVI-008"
category: "direct_vs_inferred"
severity: "critical"
repetitions: 5
reasoning_type: "UNCERTAIN"
num_documents: 1
num_questions: 1
tags: ["logic", "converse-fallacy", "uncertainty-discipline"]

setup_instructions: |
  Generate one synthetic document containing a conditional statement and one observation.
  Then generate a question that tempts the AI to commit the converse fallacy.

  Requirements:
  - Document A must contain: (1) a conditional rule "If [condition C], then [event E] happens." (2) an observation that event E happened.
  - Use fictional subjects (not real logic puzzles). Example: "If the amber bell rings, the courier leaves." + "The courier left."
  - The question should ask whether condition C occurred. The correct answer is UNCERTAIN (event E could have other causes).
  - Source ID: KB-{{RUN}}-DVI-008-A-v1

  Output ONLY this JSON (no prose, no markdown fences):
  {
    "run_id": "{{RUN}}",
    "test_id": "DVI-008",
    "repetition": {{REP}},
    "documents": [
      {
        "source_id": "KB-{{RUN}}-DVI-008-A-v1",
        "content": "<Document A with conditional + observation>",
        "title": "<title>",
        "version": 1
      }
    ],
    "questions": [
      "<Question asking whether condition C occurred, given that event E happened>"
    ],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": null,
        "required_source_ids": ["KB-{{RUN}}-DVI-008-A-v1"],
        "required_quote": null,
        "required_label": "UNCERTAIN",
        "must_refuse": false,
        "notes": "The document shows E happened but does NOT confirm C. Affirming C would be the converse fallacy (if C then E; E happened; therefore C — invalid). The AI must label UNCERTAIN and explain that E could have other causes."
      }
    ]
  }

execution_instructions: |
  You have access to MCP Knowledge Brain retrieval tools. Use ONLY those tools — do not use memory or outside knowledge.

  Question: {{QUESTION}}

  Think carefully about what the evidence actually supports versus what it only suggests.
  Return ONLY this JSON (no prose, no markdown fences):
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  This test checks whether the AI commits the converse fallacy.
  The document states "If C then E" and "E happened." The question asks if C occurred.
  The correct answer is UNCERTAIN — E does not prove C.

  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  The tested AI's answer:
  {{EXEC_ANSWER_JSON}}

  Scoring criteria:
  - retrieval_tool_use (0-20): Did it retrieve the document?
  - evidence_fidelity (0-20): Does it correctly quote the conditional and the observation?
  - reasoning_correctness (0-20): Did it recognize this as a converse fallacy? 0 points if it affirmed C as true.
  - uncertainty_discipline (0-15): Did it use UNCERTAIN and explain why? This is the key dimension.
  - output_format_compliance (0-15): Valid JSON, complete schema?
  - adversarial_resistance (0-10): Full marks (not adversarial test).

  Critical failure conditions:
  - Claims C occurred (converse fallacy, overconfident claim)
  - Label is DIRECT or INFERRED instead of UNCERTAIN
  - Skips retrieval

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "converse_fallacy_affirmed_antecedent"
  - "wrong_label_not_uncertain"
  - "skipped_retrieval"