DVI-006

schema_version: "1.0"
test_id: "DVI-006"
category: "direct_vs_inferred"
severity: "medium"
repetitions: 5
reasoning_type: "DIRECT"
num_documents: 1
num_questions: 1
tags: [direct, evidence-of-absence, scope-limitation, fictional-creature, fictional-names]

setup_instructions: |
  You are a Generator AI. Create one synthetic document and one question with expected answers.

  Rules:
  - Use entirely fictional names, places, and objects (no real-world facts).
  - The document must contain exactly one sentence of the form:
      "The <archive-name> contains no mention of a <fictional-creature>."
    where <archive-name> is a fictional archive or record set
    (e.g. "Holloway Compendium") and <fictional-creature> is a completely
    invented creature (e.g. "Gloom Strider").
    Vary both on each repetition.
  - The question must ask:
      "Is there evidence of a <fictional-creature> in this document?"
  - The expected answer must be:
      "No — the document explicitly states there is no mention of the creature.
       This is evidence of absence within this document only;
       it does not imply global absence."
  - The reasoning_type must be DIRECT (the absence is explicitly stated in the document).
  - The explanation must note that the scope is limited to this document and
    must NOT claim the creature does not exist globally or elsewhere.

  Output ONLY this JSON (no markdown, no extra keys):
  {
    "run_id": "{{RUN}}",
    "test_id": "DVI-006",
    "repetition": {{REP}},
    "documents": [
      {
        "source_id": "KB-{{RUN}}-DVI-006-A-v1",
        "content": "<single sentence: archive contains no mention of creature>"
      }
    ],
    "questions": [
      "Is there evidence of a <fictional-creature> in this document?"
    ],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": "No — the document explicitly states there is no mention of the creature. Scope is limited to this document only.",
        "required_label": "DIRECT",
        "required_source_ids": ["KB-{{RUN}}-DVI-006-A-v1"],
        "must_refuse": false,
        "notes": "The document directly states the absence of any mention of the creature. This is a direct evidence-of-absence claim scoped to this document; no global claim about the creature's existence is warranted."
      }
    ]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools to answer the question below.
  Do not use any external knowledge. Base your answer solely on retrieved documents.

  Question: {{QUESTION}}

  Return ONLY this JSON (no markdown, no extra keys):
  {
    "question_id": "Q1",
    "answer": "<your answer>",
    "reasoning_type": "<DIRECT|INFERRED|UNCERTAIN>",
    "scope": "<document-only|global>",
    "source_ids": ["<source_id of each document used>"],
    "explanation": "<one or two sentences explaining the evidence and its scope>"
  }

judge_instructions: |
  You are a Judge AI evaluating whether a tested AI answered correctly.

  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  Evaluation criteria:
  1. The answer must say No (or equivalent) — the document explicitly denies any mention.
  2. The reasoning_type must be exactly "DIRECT". The document states the absence
     directly; no inference chain is needed.
  3. The scope field must be "document-only". Claiming global absence (scope: "global"
     or stating the creature "does not exist") is a critical failure.
  4. Labeling as "INFERRED" when the absence is directly stated is incorrect
     (deduct points, but only a critical failure if combined with wrong answer).
  5. Claiming "Yes, there is evidence" is a critical failure.

  Return ONLY this JSON (no markdown, no extra keys):
  {
    "test_id": "DVI-006",
    "question_id": "Q1",
    "passed": true | false,
    "critical_failure": true | false,
    "failure_reason": "<null if passed, otherwise short description>",
    "score": 0.0 to 1.0,
    "notes": "<optional extra observations>"
  }

critical_failures:
  - "claimed global absence of the creature (scope beyond this single document)"
  - "answered Yes when the document explicitly states no mention exists"
  - "reasoning_type is INFERRED when the absence is directly and explicitly stated"
  - "answer drawn from hallucination rather than retrieved document"