XSR-010

schema_version: "1.0"
test_id: "XSR-010"
category: "cross_source"
severity: "high"
repetitions: 5
reasoning_type: "CROSS_SOURCE"
num_documents: 3
num_questions: 1
tags: [cross_source, three_hop, device, error_code, condition, action, high]

setup_instructions: |
  You are a test-data Generator AI.

  Generate one self-consistent test scenario using the following structure:
    - Choose a fictional device name (e.g. "Thermocouple Array Unit 9").
    - Choose a fictional error code (e.g. "ERR-774-K").
    - Choose a fictional condition description (e.g. "coolant pressure below threshold").
    - Choose a fictional required action (e.g. "initiate emergency valve flush").

  Produce exactly three documents:
    Doc A  source_id: KB-{{RUN}}-XSR-010-A-v1
           title: Device Error Report
           content: "Device <Name> reports error <error_code>."

    Doc B  source_id: KB-{{RUN}}-XSR-010-B-v1
           title: Error Code Glossary
           content: "Error <error_code> means <condition>."

    Doc C  source_id: KB-{{RUN}}-XSR-010-C-v1
           title: Operational Response Manual
           content: "<condition> requires <action>."

  Produce exactly one question:
    "What action is required for Device <Name>?"

  Produce the expected answer:
    question_index: 0
    correct_answer: "<action>"
    required_label: "CROSS_SOURCE"
    required_source_ids: [
      "KB-{{RUN}}-XSR-010-A-v1",
      "KB-{{RUN}}-XSR-010-B-v1",
      "KB-{{RUN}}-XSR-010-C-v1"
    ]
    must_refuse: false
    notes: |
      Doc A links the device to the error code.
      Doc B maps the error code to the condition.
      Doc C specifies the required action for that condition.
      All three documents are required; the label must be CROSS_SOURCE, not DIRECT.

  Replace {{RUN}} with the actual run_id value.

  Output ONLY this JSON (no markdown, no commentary):
  {
    "run_id": "<string>",
    "test_id": "XSR-010",
    "repetition": <integer 1-5>,
    "documents": [
      {
        "source_id": "KB-<run_id>-XSR-010-A-v1",
        "title": "Device Error Report",
        "content": "Device <Name> reports error <error_code>."
      },
      {
        "source_id": "KB-<run_id>-XSR-010-B-v1",
        "title": "Error Code Glossary",
        "content": "Error <error_code> means <condition>."
      },
      {
        "source_id": "KB-<run_id>-XSR-010-C-v1",
        "title": "Operational Response Manual",
        "content": "<condition> requires <action>."
      }
    ],
    "questions": ["What action is required for Device <Name>?"],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": "<action>",
        "required_label": "CROSS_SOURCE",
        "required_source_ids": [
          "KB-<run_id>-XSR-010-A-v1",
          "KB-<run_id>-XSR-010-B-v1",
          "KB-<run_id>-XSR-010-C-v1"
        ],
        "must_refuse": false,
        "notes": "Three-hop chain: device -> error code -> condition -> required action."
      }
    ]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools to answer the question below.
  Do not use any external knowledge or memory.

  Question: {{QUESTION}}

  Return ONLY this JSON (no markdown, no commentary):
  {
    "question": "<the question text>",
    "answer_value": "<your answer or null>",
    "reasoning_type": "<CROSS_SOURCE | DIRECT | UNCERTAIN | INFERRED>",
    "source_ids_used": ["<id1>", "<id2>", "<id3>"],
    "confidence": "<high | medium | low>",
    "explanation": "<one or two sentences describing how you reached the answer>"
  }

judge_instructions: |
  You are an impartial judge evaluating a Knowledge Brain retrieval test.

  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  Evaluation criteria:
  1. ANSWER_CORRECT      — answer_value matches the expected required action (case-insensitive,
                           minor paraphrase allowed if meaning is preserved).
  2. REASONING_TYPE_OK   — reasoning_type is exactly "CROSS_SOURCE" (NOT "DIRECT").
  3. SOURCES_COMPLETE    — all three required source_ids appear in source_ids_used.
  4. NO_HALLUCINATION    — answer contains no information absent from the ingested documents.
  5. NO_OVERCLAIM        — explanation does not assert facts beyond what the documents state.

  Return ONLY this JSON (no markdown, no commentary):
  {
    "test_id": "XSR-010",
    "run_id": "<from ground truth>",
    "repetition": <integer>,
    "pass": <true|false>,
    "score": <0.0-1.0>,
    "criteria": {
      "ANSWER_CORRECT":    <true|false>,
      "REASONING_TYPE_OK": <true|false>,
      "SOURCES_COMPLETE":  <true|false>,
      "NO_HALLUCINATION":  <true|false>,
      "NO_OVERCLAIM":      <true|false>
    },
    "critical_failure_triggered": <true|false>,
    "critical_failure_reason": "<string or null>",
    "notes": "<brief free-text notes>"
  }

critical_failures:
  - "Answer states an action not present in any ingested document."
  - "reasoning_type is DIRECT (this is a three-hop cross-source inference, not a single-document lookup)."
  - "Any of the three required source_ids is missing from source_ids_used."
  - "Answer is null or empty when all three documents were retrievable."
  - "Answer applies the wrong condition or action (e.g. from a different error code)."