DVI-002

schema_version: "1.0"
test_id: "DVI-002"
category: "direct_vs_inferred"
severity: "high"
repetitions: 5
reasoning_type: "INFERRED"
num_documents: 1
num_questions: 1
tags: [inferred, two-premise, conditional, state-inference, fictional-names]

setup_instructions: |
  You are a Generator AI. Create one synthetic document and one question with expected answers.

  Rules:
  - Use entirely fictional names, places, and objects (no real-world facts).
  - The document must contain exactly two sentences:
      Sentence 1: "The <container> is open only when the <indicator> is on."
      Sentence 2: "The <indicator> is on."
    where <container> is a fictional container (e.g. "Velox Chamber"),
    and <indicator> is a fictional indicator (e.g. "Prism Light").
    Vary both names on each repetition.
  - The question must ask: "Is the <container> open?"
  - The expected answer must be "Yes" and the reasoning_type must be INFERRED.
    Explanation: the document never directly says the container is open;
    the answer requires combining both sentences (conditional + state).
  - Do NOT allow a document that directly states "<container> is open."

  Output ONLY this JSON (no markdown, no extra keys):
  {
    "run_id": "{{RUN}}",
    "test_id": "DVI-002",
    "repetition": {{REP}},
    "documents": [
      {
        "source_id": "KB-{{RUN}}-DVI-002-A-v1",
        "content": "<sentence 1> <sentence 2>"
      }
    ],
    "questions": [
      "Is the <container> open?"
    ],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": "Yes",
        "required_label": "INFERRED",
        "required_source_ids": ["KB-{{RUN}}-DVI-002-A-v1"],
        "must_refuse": false,
        "notes": "The document states the container opens when the indicator is on, and confirms the indicator is on. The open state is inferred from combining both sentences, not stated directly."
      }
    ]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools to answer the question below.
  Do not use any external knowledge. Base your answer solely on retrieved documents.

  Question: {{QUESTION}}

  Return ONLY this JSON (no markdown, no extra keys):
  {
    "question_id": "Q1",
    "answer": "<Yes|No|Unknown>",
    "reasoning_type": "<DIRECT|INFERRED|UNCERTAIN>",
    "source_ids": ["<source_id of each document used>"],
    "explanation": "<one or two sentences describing which sentences you combined to reach the answer>"
  }

judge_instructions: |
  You are a Judge AI evaluating whether a tested AI answered correctly.

  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  Evaluation criteria:
  1. The answer field must be "Yes" (case-insensitive).
  2. The reasoning_type must be exactly "INFERRED".
  3. Labeling this as "DIRECT" is a critical failure: the document never states
     the container is open directly; the conclusion requires combining two sentences.
  4. Answering "No" or "Unknown" is a critical failure.
  5. The explanation should reference both the conditional sentence and the state
     sentence; if it only cites one, deduct points but do not critical-fail.

  Return ONLY this JSON (no markdown, no extra keys):
  {
    "test_id": "DVI-002",
    "question_id": "Q1",
    "passed": true | false,
    "critical_failure": true | false,
    "failure_reason": "<null if passed, otherwise short description>",
    "score": 0.0 to 1.0,
    "notes": "<optional extra observations>"
  }

critical_failures:
  - "reasoning_type is DIRECT (the open state is never explicitly stated; both sentences must be combined)"
  - "answer is No or Unknown when the evidence supports Yes"
  - "answer drawn from hallucination rather than retrieved document"