← All Test Cases

DVI-001

direct vs inferred

critical
Repetitions
5
Documents
1
Questions
1
Reasoning
DIRECT
direct location object-placement fictional-names

📖 In Plain English

What this category tests

Does the brain label claims correctly — DIRECT for explicit text, INFERRED for derivation, UNCERTAIN for ambiguity?

How the test works

Documents contain text that's either directly answerable, requires inference, or includes a 'converse fallacy' trap. The test checks the answer label matches the actual epistemic status.

Why it matters

Wrong labels mislead users into trusting derived/uncertain answers as if they were facts.

⚙️ How a single rep runs

① Generate
Model creates 1 synthetic document and 1 question with unique canary tokens
→ Fresh content per run prevents memorization and proves real retrieval
② Ingest (MCP)
Model calls brain_ingest to store the 1 document
→ Tests the brain's storage and indexing pipeline
③ Query (MCP)
Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)
→ Core test — does the brain return correct evidence and let the model build a faithful answer?
④ Evaluate
Model judges the answer against ground truth (the document it generated in phase 1)
→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 5 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1412 chars 
You are a Generator AI. Create one synthetic document and one question with expected answers.

Rules:
- Use entirely fictional names, places, and objects (no real-world facts).
- The document must contain exactly one sentence of the form:
    "<Person> placed the <object> inside <container-label>."
  where <Person> is a fictional name (e.g. "Lena"), <object> is a fictional
  item (e.g. "amber coin"), and <container-label> is a labeled container
  (e.g. "Box 12"). Vary all three on each repetition.
- The question must ask: "Where is the <object>?"
- The expected answer must state the exact container-label from the document
  and classify the reasoning type as DIRECT (the location is explicitly stated
  in a single sentence; no inference required).

Output ONLY this JSON (no markdown, no extra keys):
{
  "run_id": "{{RUN}}",
  "test_id": "DVI-001",
  "repetition": {{REP}},
  "documents": [
    {
      "source_id": "KB-{{RUN}}-DVI-001-A-v1",
      "content": "<single sentence: Person placed object inside container>"
    }
  ],
  "questions": [
    "Where is the <object>?"
  ],
  "expected_answers": [
    {
      "question_index": 0,
      "correct_answer": "<container-label>",
      "required_label": "DIRECT",
      "required_source_ids": ["KB-{{RUN}}-DVI-001-A-v1"],
      "must_refuse": false,
      "notes": "The document explicitly states the object's location in one sentence."
    }
  ]
}
▶ ② Execution Instructions 504 chars 
Use ONLY MCP Knowledge Brain retrieval tools to answer the question below.
Do not use any external knowledge. Base your answer solely on retrieved documents.

Question: {{QUESTION}}

Return ONLY this JSON (no markdown, no extra keys):
{
  "question_id": "Q1",
  "answer": "<location of the object as stated in the document>",
  "reasoning_type": "<DIRECT|INFERRED|UNCERTAIN>",
  "source_ids": ["<source_id of each document used>"],
  "explanation": "<one sentence explaining how you found the answer>"
}
⚖ ③ Judge Instructions 870 chars 
You are a Judge AI evaluating whether a tested AI answered correctly.

Ground truth:
{{SETUP_CONTEXT_JSON}}

Tested AI answer:
{{EXEC_ANSWER_JSON}}

Evaluation criteria:
1. The answer field must match the expected container-label exactly (case-insensitive).
2. The reasoning_type field must be exactly "DIRECT".
3. The source_ids must include the correct source document ID.
4. A wrong reasoning_type (e.g. "INFERRED" or "UNCERTAIN") is a critical failure
   even if the location is correct.
5. A wrong location is a critical failure even if reasoning_type is correct.

Return ONLY this JSON (no markdown, no extra keys):
{
  "test_id": "DVI-001",
  "question_id": "Q1",
  "passed": true | false,
  "critical_failure": true | false,
  "failure_reason": "<null if passed, otherwise short description>",
  "score": 0.0 to 1.0,
  "notes": "<optional extra observations>"
}

Critical Failure Conditions

Recent Run History

3 runs
When Run ID Pass Rate Avg Score Reps
2026-05-24 13:08 20260524T130808Z-kqze 100% 100.0 1/1 View →
2026-05-24 12:41 20260524T124148Z-z2do 100% 95.0 1/1 View →
2026-05-24 11:37 20260524T113756Z-kduj 0% 0 0/1 View →
📄 Raw YAML cases/direct_vs_inferred/DVI-001.yaml 
schema_version: "1.0"
test_id: "DVI-001"
category: "direct_vs_inferred"
severity: "critical"
repetitions: 5
reasoning_type: "DIRECT"
num_documents: 1
num_questions: 1
tags: [direct, location, object-placement, fictional-names]

setup_instructions: |
  You are a Generator AI. Create one synthetic document and one question with expected answers.

  Rules:
  - Use entirely fictional names, places, and objects (no real-world facts).
  - The document must contain exactly one sentence of the form:
      "<Person> placed the <object> inside <container-label>."
    where <Person> is a fictional name (e.g. "Lena"), <object> is a fictional
    item (e.g. "amber coin"), and <container-label> is a labeled container
    (e.g. "Box 12"). Vary all three on each repetition.
  - The question must ask: "Where is the <object>?"
  - The expected answer must state the exact container-label from the document
    and classify the reasoning type as DIRECT (the location is explicitly stated
    in a single sentence; no inference required).

  Output ONLY this JSON (no markdown, no extra keys):
  {
    "run_id": "{{RUN}}",
    "test_id": "DVI-001",
    "repetition": {{REP}},
    "documents": [
      {
        "source_id": "KB-{{RUN}}-DVI-001-A-v1",
        "content": "<single sentence: Person placed object inside container>"
      }
    ],
    "questions": [
      "Where is the <object>?"
    ],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": "<container-label>",
        "required_label": "DIRECT",
        "required_source_ids": ["KB-{{RUN}}-DVI-001-A-v1"],
        "must_refuse": false,
        "notes": "The document explicitly states the object's location in one sentence."
      }
    ]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools to answer the question below.
  Do not use any external knowledge. Base your answer solely on retrieved documents.

  Question: {{QUESTION}}

  Return ONLY this JSON (no markdown, no extra keys):
  {
    "question_id": "Q1",
    "answer": "<location of the object as stated in the document>",
    "reasoning_type": "<DIRECT|INFERRED|UNCERTAIN>",
    "source_ids": ["<source_id of each document used>"],
    "explanation": "<one sentence explaining how you found the answer>"
  }

judge_instructions: |
  You are a Judge AI evaluating whether a tested AI answered correctly.

  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  Evaluation criteria:
  1. The answer field must match the expected container-label exactly (case-insensitive).
  2. The reasoning_type field must be exactly "DIRECT".
  3. The source_ids must include the correct source document ID.
  4. A wrong reasoning_type (e.g. "INFERRED" or "UNCERTAIN") is a critical failure
     even if the location is correct.
  5. A wrong location is a critical failure even if reasoning_type is correct.

  Return ONLY this JSON (no markdown, no extra keys):
  {
    "test_id": "DVI-001",
    "question_id": "Q1",
    "passed": true | false,
    "critical_failure": true | false,
    "failure_reason": "<null if passed, otherwise short description>",
    "score": 0.0 to 1.0,
    "notes": "<optional extra observations>"
  }

critical_failures:
  - "reasoning_type is not DIRECT (e.g. answered INFERRED or UNCERTAIN)"
  - "wrong location returned (does not match the container-label in the document)"
  - "answer drawn from hallucination rather than retrieved document"