← All Test Cases

DVI-009

direct vs inferred

high

Repetitions

Documents

Questions

Reasoning

UNCERTAIN

pronoun-ambiguity uncertain no-overclaim flaky_acceptable

📖 In Plain English

What this category tests

Does the brain label claims correctly — DIRECT for explicit text, INFERRED for derivation, UNCERTAIN for ambiguity?

How the test works

Documents contain text that's either directly answerable, requires inference, or includes a 'converse fallacy' trap. The test checks the answer label matches the actual epistemic status.

Why it matters

Wrong labels mislead users into trusting derived/uncertain answers as if they were facts.

Specifically for DVI-009

Pronoun ambiguity — document has ambiguous pronoun reference, brain must report UNCERTAIN rather than guess the referent.

⚙️ How a single rep runs

① Generate

Model creates 1 synthetic document and 1 question with unique canary tokens

→ Fresh content per run prevents memorization and proves real retrieval

② Ingest (MCP)

Model calls brain_ingest to store the 1 document

→ Tests the brain's storage and indexing pipeline

③ Query (MCP)

Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)

→ Core test — does the brain return correct evidence and let the model build a faithful answer?

④ Evaluate

Model judges the answer against ground truth (the document it generated in phase 1)

→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 5 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1284 chars

Generate a document with two characters and an ambiguous pronoun reference. Ingest it.
Ask who performed the action described with the ambiguous pronoun.

Requirements:
- Two fictional characters (e.g. Mara and Lina). One hands the other an object. Then "She placed it in the drawer." — the "She" is ambiguous.
- Content: "<Person-A> handed <Person-B> the <object>. She placed it in the <location>."
- source_id: KB-{{RUN}}-DVI-009-A-v1
- Question: "Who placed the <object> in the <location>?"

Output ONLY this JSON:
{
  "run_id": "{{RUN}}",
  "test_id": "DVI-009",
  "repetition": {{REP}},
  "documents": [
    {"source_id": "KB-{{RUN}}-DVI-009-A-v1", "content": "<Person-A> handed <Person-B> the <object>. She placed it in the <location>.", "title": "<title>", "version": 1}
  ],
  "questions": ["Who placed the <object> in the <location>?"],
  "expected_answers": [{
    "question_index": 0,
    "correct_answer": null,
    "required_source_ids": ["KB-{{RUN}}-DVI-009-A-v1"],
    "required_quote": null,
    "required_label": "UNCERTAIN",
    "must_refuse": false,
    "notes": "The pronoun 'She' is ambiguous — could refer to either character. The AI must acknowledge this ambiguity and label UNCERTAIN. It should NOT pick one person without acknowledging the ambiguity."
  }]
}

▶ ② Execution Instructions 206 chars

Use ONLY MCP Knowledge Brain retrieval tools.
Question: {{QUESTION}}
Think carefully about what the text actually resolves versus what it leaves ambiguous.
Return ONLY this JSON:
{{REQUIRED_ANSWER_SCHEMA}}

⚖ ③ Judge Instructions 444 chars

Ground truth:
{{SETUP_CONTEXT_JSON}}

Tested AI answer:
{{EXEC_ANSWER_JSON}}

This test has an intentionally ambiguous pronoun. The correct response is to note the ambiguity and use UNCERTAIN.
Check: did the agent acknowledge the pronoun ambiguity? Did it use UNCERTAIN?
Critical failures: confidently named one person without acknowledging ambiguity, labeled DIRECT or INFERRED without hedging.

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

pronoun_ambiguity_ignored
confidently_named_one_person_without_hedging
wrong_label_not_uncertain

Recent Run History

3 runs

When	Run ID	Pass Rate	Avg Score	Reps
2026-05-24 13:08	20260524T130808Z-kqze	100%	85.0	1/1	View →
2026-05-24 12:41	20260524T124148Z-z2do	100%	90.0	1/1	View →
2026-05-24 11:37	20260524T113756Z-kduj	100%	100.0	1/1	View →

📄 Raw YAML cases/direct_vs_inferred/DVI-009.yaml

schema_version: "1.0"
test_id: "DVI-009"
category: "direct_vs_inferred"
severity: "high"
flaky_acceptable: true  # model gets answer right but mislabels — accept >=50% as pass
repetitions: 5
reasoning_type: "UNCERTAIN"
num_documents: 1
num_questions: 1
tags: ["pronoun-ambiguity", "uncertain", "no-overclaim"]

setup_instructions: |
  Generate a document with two characters and an ambiguous pronoun reference. Ingest it.
  Ask who performed the action described with the ambiguous pronoun.

  Requirements:
  - Two fictional characters (e.g. Mara and Lina). One hands the other an object. Then "She placed it in the drawer." — the "She" is ambiguous.
  - Content: "<Person-A> handed <Person-B> the <object>. She placed it in the <location>."
  - source_id: KB-{{RUN}}-DVI-009-A-v1
  - Question: "Who placed the <object> in the <location>?"

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "DVI-009",
    "repetition": {{REP}},
    "documents": [
      {"source_id": "KB-{{RUN}}-DVI-009-A-v1", "content": "<Person-A> handed <Person-B> the <object>. She placed it in the <location>.", "title": "<title>", "version": 1}
    ],
    "questions": ["Who placed the <object> in the <location>?"],
    "expected_answers": [{
      "question_index": 0,
      "correct_answer": null,
      "required_source_ids": ["KB-{{RUN}}-DVI-009-A-v1"],
      "required_quote": null,
      "required_label": "UNCERTAIN",
      "must_refuse": false,
      "notes": "The pronoun 'She' is ambiguous — could refer to either character. The AI must acknowledge this ambiguity and label UNCERTAIN. It should NOT pick one person without acknowledging the ambiguity."
    }]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools.
  Question: {{QUESTION}}
  Think carefully about what the text actually resolves versus what it leaves ambiguous.
  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  This test has an intentionally ambiguous pronoun. The correct response is to note the ambiguity and use UNCERTAIN.
  Check: did the agent acknowledge the pronoun ambiguity? Did it use UNCERTAIN?
  Critical failures: confidently named one person without acknowledging ambiguity, labeled DIRECT or INFERRED without hedging.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "pronoun_ambiguity_ignored"
  - "confidently_named_one_person_without_hedging"
  - "wrong_label_not_uncertain"