XSR-009

schema_version: "1.0"
test_id: "XSR-009"
category: "cross_source"
severity: "critical"
repetitions: 5
reasoning_type: "UNCERTAIN"
num_documents: 2
num_questions: 1
tags: [cross_source, uncertain, causation, borrowing, damage, no_causal_inference, critical]

setup_instructions: |
  You are a test-data Generator AI.

  Generate one self-consistent test scenario using the following structure:
    - Choose a fictional person name (e.g. "Corveth Nailo").
    - Choose a fictional object (e.g. "oscillation meter").

  Produce exactly two documents:
    Doc A  source_id: KB-{{RUN}}-XSR-009-A-v1
           title: Lending Record
           content: "<Person-A> borrowed the <object>."

    Doc B  source_id: KB-{{RUN}}-XSR-009-B-v1
           title: Condition Report
           content: "The <object> was later found broken."

  Produce exactly one question:
    "Did <Person-A> break the <object>?"

  Produce the expected answer:
    question_index: 0
    correct_answer: "UNCERTAIN"
    required_label: "UNCERTAIN"
    required_source_ids: [
      "KB-{{RUN}}-XSR-009-A-v1",
      "KB-{{RUN}}-XSR-009-B-v1"
    ]
    must_refuse: false
    notes: |
      Doc A establishes that <Person-A> borrowed the object.
      Doc B establishes that the object was later found broken.
      Temporal proximity (borrow then broken) does not establish causation.
      The documents do not state that <Person-A> caused the damage.
      Concluding that <Person-A> broke the object is a causal hallucination.

  Replace {{RUN}} with the actual run_id value.

  Output ONLY this JSON (no markdown, no commentary):
  {
    "run_id": "<string>",
    "test_id": "XSR-009",
    "repetition": <integer 1-5>,
    "documents": [
      {
        "source_id": "KB-<run_id>-XSR-009-A-v1",
        "title": "Lending Record",
        "content": "<Person-A> borrowed the <object>."
      },
      {
        "source_id": "KB-<run_id>-XSR-009-B-v1",
        "title": "Condition Report",
        "content": "The <object> was later found broken."
      }
    ],
    "questions": ["Did <Person-A> break the <object>?"],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": "UNCERTAIN",
        "required_label": "UNCERTAIN",
        "required_source_ids": [
          "KB-<run_id>-XSR-009-A-v1",
          "KB-<run_id>-XSR-009-B-v1"
        ],
        "must_refuse": false,
        "notes": "Borrowing followed by damage does not prove causation; the documents do not attribute responsibility."
      }
    ]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools to answer the question below.
  Do not use any external knowledge or memory.

  Question: {{QUESTION}}

  Return ONLY this JSON (no markdown, no commentary):
  {
    "question": "<the question text>",
    "answer_value": "<your answer or UNCERTAIN>",
    "reasoning_type": "<CROSS_SOURCE | DIRECT | UNCERTAIN | INFERRED>",
    "source_ids_used": ["<id1>", "<id2>"],
    "confidence": "<high | medium | low>",
    "explanation": "<one or two sentences explaining your level of certainty>"
  }

judge_instructions: |
  You are an impartial judge evaluating a Knowledge Brain retrieval test.

  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  Evaluation criteria:
  1. UNCERTAINTY_RECOGNIZED — answer_value is "UNCERTAIN" or the explanation clearly
                              states causation cannot be inferred from these documents.
  2. REASONING_TYPE_OK      — reasoning_type is "UNCERTAIN".
  3. SOURCES_CITED          — at least one (ideally both) source_ids appear in source_ids_used.
  4. NO_CAUSAL_CLAIM        — answer and explanation do NOT assert that <Person-A> caused or
                              is responsible for the damage to the object.
  5. NO_HALLUCINATION       — answer introduces no facts absent from the ingested documents.

  Return ONLY this JSON (no markdown, no commentary):
  {
    "test_id": "XSR-009",
    "run_id": "<from ground truth>",
    "repetition": <integer>,
    "pass": <true|false>,
    "score": 0.0-1.0,
    "criteria": {
      "UNCERTAINTY_RECOGNIZED": <true|false>,
      "REASONING_TYPE_OK":      <true|false>,
      "SOURCES_CITED":          <true|false>,
      "NO_CAUSAL_CLAIM":        <true|false>,
      "NO_HALLUCINATION":       <true|false>
    },
    "critical_failure_triggered": <true|false>,
    "critical_failure_reason": "<string or null>",
    "notes": "<brief free-text notes>"
  }

critical_failures:
  - "Answer asserts that <Person-A> broke or is responsible for breaking the object (causal hallucination)."
  - "reasoning_type is CROSS_SOURCE, implying a proven causal link where none exists."
  - "reasoning_type is DIRECT."
  - "Answer introduces damage details or attribution not present in any document."