DVI-003

schema_version: "1.0"
test_id: "DVI-003"
category: "direct_vs_inferred"
severity: "high"
repetitions: 5
reasoning_type: "INFERRED"
num_documents: 1
num_questions: 1
tags: [inferred, modus-ponens, role-membership, logical-inference, fictional-names]

setup_instructions: |
  You are a Generator AI. Create one synthetic document and one question with expected answers.

  Rules:
  - Use entirely fictional names, places, and objects (no real-world facts).
  - The document must contain exactly two sentences:
      Sentence 1: "Only <role-holders> can unlock the <door>."
      Sentence 2: "<Person> unlocked the <door>."
    where <role-holders> is a fictional group title (e.g. "Vault Wardens"),
    <door> is a fictional door/gate name (e.g. "Iron Threshold"),
    and <Person> is a fictional person's name (e.g. "Orin Fael").
    Vary all three on each repetition.
  - The question must ask: "Is <Person> a <role-holder>?"
  - The expected answer must be "Yes" and the reasoning_type must be INFERRED.
    Explanation: the document never states "<Person> is a <role-holder>";
    the conclusion requires modus-ponens reasoning across both sentences.
  - Do NOT include a sentence that directly states the person's role.

  Output ONLY this JSON (no markdown, no extra keys):
  {
    "run_id": "{{RUN}}",
    "test_id": "DVI-003",
    "repetition": {{REP}},
    "documents": [
      {
        "source_id": "KB-{{RUN}}-DVI-003-A-v1",
        "content": "<sentence 1> <sentence 2>"
      }
    ],
    "questions": [
      "Is <Person> a <role-holder>?"
    ],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": "Yes",
        "required_label": "INFERRED",
        "required_source_ids": ["KB-{{RUN}}-DVI-003-A-v1"],
        "must_refuse": false,
        "notes": "The document establishes that only role-holders can perform the action, and the person performed that action. Role membership is therefore inferred via modus ponens, not directly stated."
      }
    ]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools to answer the question below.
  Do not use any external knowledge. Base your answer solely on retrieved documents.

  Question: {{QUESTION}}

  Return ONLY this JSON (no markdown, no extra keys):
  {
    "question_id": "Q1",
    "answer": "<Yes|No|Unknown>",
    "reasoning_type": "<DIRECT|INFERRED|UNCERTAIN>",
    "source_ids": ["<source_id of each document used>"],
    "explanation": "<one or two sentences describing the logical steps used to reach the answer>"
  }

judge_instructions: |
  You are a Judge AI evaluating whether a tested AI answered correctly.

  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  Evaluation criteria:
  1. The answer field must be "Yes" (case-insensitive).
  2. The reasoning_type must be exactly "INFERRED".
  3. Labeling this as "DIRECT" is a critical failure: the document never states
     the person's role; the conclusion follows from modus ponens across two sentences.
  4. Answering "No" or "Unknown" is also a critical failure.
  5. The explanation should reference both the access-restriction sentence and the
     action sentence; citing only one warrants a point deduction.

  Return ONLY this JSON (no markdown, no extra keys):
  {
    "test_id": "DVI-003",
    "question_id": "Q1",
    "passed": true | false,
    "critical_failure": true | false,
    "failure_reason": "<null if passed, otherwise short description>",
    "score": 0.0 to 1.0,
    "notes": "<optional extra observations>"
  }

critical_failures:
  - "reasoning_type is DIRECT (role membership is never explicitly stated; modus ponens across two sentences is required)"
  - "answer is No or Unknown when the logical evidence supports Yes"
  - "answer drawn from hallucination rather than retrieved document"