DVI-007

schema_version: "1.0"
test_id: "DVI-007"
category: "direct_vs_inferred"
severity: "medium"
repetitions: 5
reasoning_type: "DIRECT"
num_documents: 1
num_questions: 1
tags: [direct, two-state, selection, gate-door, fictional-names]

setup_instructions: |
  You are a Generator AI. Create one synthetic document and one question with expected answers.

  Rules:
  - Use entirely fictional names, places, and objects (no real-world facts).
  - The document must contain exactly two sentences:
      Sentence 1: "The <gate-A> was locked."
      Sentence 2: "The <gate-B> was open."
    where <gate-A> and <gate-B> are distinct fictional gate or door names
    (e.g. "Dusk Gate" and "Ember Door"). Vary both on each repetition.
  - The question must ask: "Which gate (or door) was open?"
  - The expected answer must be <gate-B> exactly and the reasoning_type must be DIRECT.
    The document states it directly in sentence 2; no reasoning chain is needed.
  - Do NOT reverse the states or make them ambiguous.

  Output ONLY this JSON (no markdown, no extra keys):
  {
    "run_id": "{{RUN}}",
    "test_id": "DVI-007",
    "repetition": {{REP}},
    "documents": [
      {
        "source_id": "KB-{{RUN}}-DVI-007-A-v1",
        "content": "<sentence 1> <sentence 2>"
      }
    ],
    "questions": [
      "Which gate (or door) was open?"
    ],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": "<gate-B name>",
        "required_label": "DIRECT",
        "required_source_ids": ["KB-{{RUN}}-DVI-007-A-v1"],
        "must_refuse": false,
        "notes": "The document explicitly states which gate was open in a single sentence. No inference is required."
      }
    ]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools to answer the question below.
  Do not use any external knowledge. Base your answer solely on retrieved documents.

  Question: {{QUESTION}}

  Return ONLY this JSON (no markdown, no extra keys):
  {
    "question_id": "Q1",
    "answer": "<name of the gate or door that was open>",
    "reasoning_type": "<DIRECT|INFERRED|UNCERTAIN>",
    "source_ids": ["<source_id of each document used>"],
    "explanation": "<one sentence citing the exact text that states the gate was open>"
  }

judge_instructions: |
  You are a Judge AI evaluating whether a tested AI answered correctly.

  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  Evaluation criteria:
  1. The answer must match the name of gate-B (the open gate) exactly
     (case-insensitive). Returning gate-A (the locked gate) is a critical failure.
  2. The reasoning_type must be exactly "DIRECT". Labeling this as "INFERRED"
     is a critical failure: the open state of gate-B is stated explicitly in
     a single sentence and requires no reasoning chain.
  3. The explanation should cite or closely paraphrase the sentence that names
     the open gate; if it does not, deduct points.
  4. Answering "Unknown" or returning both gate names without selecting one
     is a critical failure.

  Return ONLY this JSON (no markdown, no extra keys):
  {
    "test_id": "DVI-007",
    "question_id": "Q1",
    "passed": true | false,
    "critical_failure": true | false,
    "failure_reason": "<null if passed, otherwise short description>",
    "score": 0.0 to 1.0,
    "notes": "<optional extra observations>"
  }

critical_failures:
  - "returned the wrong gate (gate-A, the locked one, instead of gate-B)"
  - "reasoning_type is INFERRED when the open gate is directly and explicitly named"
  - "answered Unknown or failed to select a single gate"
  - "answer drawn from hallucination rather than retrieved document"