DVI-005

schema_version: "1.0"
test_id: "DVI-005"
category: "direct_vs_inferred"
severity: "high"
repetitions: 5
reasoning_type: "INFERRED"
num_documents: 1
num_questions: 1
tags: [inferred, badge-role, possession-vs-identity, ambiguity, fictional-names]

setup_instructions: |
  You are a Generator AI. Create one synthetic document and one question with expected answers.

  Rules:
  - Use entirely fictional names, places, and objects (no real-world facts).
  - The document must contain exactly two sentences:
      Sentence 1: "The <color> badge belongs to a <role>."
      Sentence 2: "<Person> carries the <color> badge."
    where <color> is a fictional color or descriptive label (e.g. "cerulean"),
    <role> is a fictional occupational title (e.g. "Drift Keeper"),
    and <Person> is a fictional name (e.g. "Sael Moran").
    Vary all three on each repetition.
  - The question must ask: "Is <Person> a <role>?"
  - The expected answer must acknowledge that the evidence strongly suggests Yes
    but must note the ambiguity: carrying the badge is not the same as owning it,
    so the identity claim is not definitively established.
  - The reasoning_type must be INFERRED.
  - The explanation in expected_answers must explicitly mention the distinction
    between badge ownership and badge carrying.
  - Do NOT include any sentence that directly states the person's role or identity.

  Output ONLY this JSON (no markdown, no extra keys):
  {
    "run_id": "{{RUN}}",
    "test_id": "DVI-005",
    "repetition": {{REP}},
    "documents": [
      {
        "source_id": "KB-{{RUN}}-DVI-005-A-v1",
        "content": "<sentence 1> <sentence 2>"
      }
    ],
    "questions": [
      "Is <Person> a <role>?"
    ],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": "Likely yes, but not certain — carrying the badge suggests the role, though ownership vs. carrying creates ambiguity.",
        "required_label": "INFERRED",
        "required_source_ids": ["KB-{{RUN}}-DVI-005-A-v1"],
        "must_refuse": false,
        "notes": "The badge belongs to a role-holder and the person carries it, making the role likely. However, the document does not confirm ownership; carrying alone does not definitively establish identity as a role-holder."
      }
    ]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools to answer the question below.
  Do not use any external knowledge. Base your answer solely on retrieved documents.

  Question: {{QUESTION}}

  Return ONLY this JSON (no markdown, no extra keys):
  {
    "question_id": "Q1",
    "answer": "<your answer, including any ambiguity you identify>",
    "reasoning_type": "<DIRECT|INFERRED|UNCERTAIN>",
    "ambiguity_noted": true | false,
    "source_ids": ["<source_id of each document used>"],
    "explanation": "<one or two sentences describing the inference and any ambiguity>"
  }

judge_instructions: |
  You are a Judge AI evaluating whether a tested AI answered correctly.

  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  Evaluation criteria:
  1. The reasoning_type must be exactly "INFERRED". Labeling this as "DIRECT"
     is a critical failure: the person's role is never stated explicitly.
  2. The answer must acknowledge that the conclusion is probable but not certain,
     and must specifically note the distinction between carrying and owning the badge.
     Failing to mention this ambiguity (ambiguity_noted == false, or no mention
     in explanation) is a critical failure.
  3. Answering a flat "Yes" with no ambiguity noted is a critical failure.
  4. Answering "No" when the evidence suggests the role is likely is a critical failure.
  5. A well-hedged "Likely yes, but not certain" with explanation earns full score.

  Return ONLY this JSON (no markdown, no extra keys):
  {
    "test_id": "DVI-005",
    "question_id": "Q1",
    "passed": true | false,
    "critical_failure": true | false,
    "failure_reason": "<null if passed, otherwise short description>",
    "score": 0.0 to 1.0,
    "notes": "<optional extra observations>"
  }

critical_failures:
  - "reasoning_type is DIRECT (the role is never explicitly stated)"
  - "no mention of the ownership-vs-carrying ambiguity (ambiguity_noted is false or absent from explanation)"
  - "flat Yes with no hedging or acknowledgment of ambiguity"
  - "answer is No when badge evidence suggests the role is probable"
  - "answer drawn from hallucination rather than retrieved document"