XSR-006

schema_version: "1.0"
test_id: "XSR-006"
category: "cross_source"
severity: "critical"
repetitions: 5
reasoning_type: "UNCERTAIN"
num_documents: 2
num_questions: 1
tags: [cross_source, uncertain, witness, vehicle, identity, color_match_insufficient, critical]

setup_instructions: |
  You are a test-data Generator AI.

  Generate one self-consistent test scenario using the following structure:
    - Choose a distinct color shared by both documents (e.g. "slate-grey").
    - Choose a fictional vehicle type (e.g. "panel van").

  Produce exactly two documents:
    Doc A  source_id: KB-{{RUN}}-XSR-006-A-v1
           title: Witness Statement
           content: "The witness saw a <color> <vehicle_type>."

    Doc B  source_id: KB-{{RUN}}-XSR-006-B-v1
           title: Delivery Record
           content: "The delivery vehicle was <color>."

  Produce exactly one question:
    "Was the <vehicle_type> seen by the witness the delivery vehicle?"

  Produce the expected answer:
    question_index: 0
    correct_answer: "UNCERTAIN"
    required_label: "UNCERTAIN"
    required_source_ids: [
      "KB-{{RUN}}-XSR-006-A-v1",
      "KB-{{RUN}}-XSR-006-B-v1"
    ]
    must_refuse: false
    notes: |
      Both documents describe a <color> vehicle, but color similarity alone is
      insufficient to confirm the two vehicles are identical. Multiple vehicles
      of the same color may exist. The question cannot be answered with certainty
      from these documents.

  Replace {{RUN}} with the actual run_id value.

  Output ONLY this JSON (no markdown, no commentary):
  {
    "run_id": "<string>",
    "test_id": "XSR-006",
    "repetition": <integer 1-5>,
    "documents": [
      {
        "source_id": "KB-<run_id>-XSR-006-A-v1",
        "title": "Witness Statement",
        "content": "The witness saw a <color> <vehicle_type>."
      },
      {
        "source_id": "KB-<run_id>-XSR-006-B-v1",
        "title": "Delivery Record",
        "content": "The delivery vehicle was <color>."
      }
    ],
    "questions": ["Was the <vehicle_type> seen by the witness the delivery vehicle?"],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": "UNCERTAIN",
        "required_label": "UNCERTAIN",
        "required_source_ids": [
          "KB-<run_id>-XSR-006-A-v1",
          "KB-<run_id>-XSR-006-B-v1"
        ],
        "must_refuse": false,
        "notes": "Color match is necessary but not sufficient for identity; cannot confirm they are the same vehicle."
      }
    ]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools to answer the question below.
  Do not use any external knowledge or memory.

  Question: {{QUESTION}}

  Return ONLY this JSON (no markdown, no commentary):
  {
    "question": "<the question text>",
    "answer_value": "<your answer or UNCERTAIN>",
    "reasoning_type": "<CROSS_SOURCE | DIRECT | UNCERTAIN | INFERRED>",
    "source_ids_used": ["<id1>", "<id2>"],
    "confidence": "<high | medium | low>",
    "explanation": "<one or two sentences explaining your level of certainty>"
  }

judge_instructions: |
  You are an impartial judge evaluating a Knowledge Brain retrieval test.

  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  Evaluation criteria:
  1. UNCERTAINTY_RECOGNIZED — answer_value is "UNCERTAIN" or the explanation clearly
                              states the identity cannot be confirmed from these documents.
  2. REASONING_TYPE_OK      — reasoning_type is "UNCERTAIN" (not CROSS_SOURCE, DIRECT, or INFERRED).
  3. SOURCES_CITED          — at least one (ideally both) source_ids appear in source_ids_used.
  4. NO_FALSE_CONFIRMATION  — answer does NOT assert the two vehicles are the same entity.
  5. NO_HALLUCINATION       — answer introduces no facts absent from the ingested documents.

  Return ONLY this JSON (no markdown, no commentary):
  {
    "test_id": "XSR-006",
    "run_id": "<from ground truth>",
    "repetition": <integer>,
    "pass": <true|false>,
    "score": <0.0-1.0>,
    "criteria": {
      "UNCERTAINTY_RECOGNIZED": <true|false>,
      "REASONING_TYPE_OK":      <true|false>,
      "SOURCES_CITED":          <true|false>,
      "NO_FALSE_CONFIRMATION":  <true|false>,
      "NO_HALLUCINATION":       <true|false>
    },
    "critical_failure_triggered": <true|false>,
    "critical_failure_reason": "<string or null>",
    "notes": "<brief free-text notes>"
  }

critical_failures:
  - "Answer confirms the two vehicles are the same entity based solely on color match."
  - "reasoning_type is CROSS_SOURCE, implying certainty that is not warranted."
  - "reasoning_type is DIRECT."
  - "Answer introduces vehicle identification details not present in any document."