XSR-001

schema_version: "1.0"
test_id: "XSR-001"
category: "cross_source"
severity: "critical"
repetitions: 5
reasoning_type: "CROSS_SOURCE"
num_documents: 2
num_questions: 1
tags: ["multi-doc", "join", "entity-property"]

setup_instructions: |
  Generate two synthetic documents forming a two-step reasoning chain, then ingest both.

  Requirements:
  - Document A: States that a fictional object with a unique code/ID IS a second fictional object. Example: "The item with code Q7 is the cobalt key."
  - Document B: States a property of that second object. Example: "The cobalt key opens Drawer 3."
  - Use fictional names/codes only. Do not use real-world facts.
  - Source IDs: KB-{{RUN}}-XSR-001-A-v1 and KB-{{RUN}}-XSR-001-B-v1
  - Generate one question that can only be answered by combining both documents. The question should ask about the property of the original code, forcing a two-document join.

  Output ONLY this JSON (no prose, no markdown fences):
  {
    "run_id": "{{RUN}}",
    "test_id": "XSR-001",
    "repetition": {{REP}},
    "documents": [
      {
        "source_id": "KB-{{RUN}}-XSR-001-A-v1",
        "content": "<Document A content>",
        "title": "<title>",
        "version": 1
      },
      {
        "source_id": "KB-{{RUN}}-XSR-001-B-v1",
        "content": "<Document B content>",
        "title": "<title>",
        "version": 1
      }
    ],
    "questions": [
      "<Question requiring both documents to answer>"
    ],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": "<the property from Document B>",
        "required_source_ids": ["KB-{{RUN}}-XSR-001-A-v1", "KB-{{RUN}}-XSR-001-B-v1"],
        "required_quote": null,
        "required_label": "CROSS_SOURCE",
        "must_refuse": false,
        "notes": "Answer must cite both source IDs. Label must be CROSS_SOURCE, not DIRECT or INFERRED."
      }
    ]
  }

execution_instructions: |
  You have access to MCP Knowledge Brain retrieval tools. Use ONLY those tools — do not use memory or outside knowledge.

  Question: {{QUESTION}}

  You must retrieve from the brain and combine information from multiple sources to answer.
  Return ONLY this JSON (no prose, no markdown fences):
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  You generated the following test data and ingested it into a knowledge brain.
  A separate AI then answered a question requiring it to combine two documents.
  Evaluate whether it found and used both documents correctly.

  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  The tested AI's answer:
  {{EXEC_ANSWER_JSON}}

  Scoring criteria:
  - retrieval_tool_use (0-20): Did it retrieve BOTH required documents? Partial retrieval = partial credit.
  - evidence_fidelity (0-20): Are both source_ids cited correctly? Quotes accurate?
  - reasoning_correctness (0-20): Is the final answer correct? Is the join logic valid?
  - uncertainty_discipline (0-15): Did it avoid fabricating a join that isn't supported?
  - output_format_compliance (0-15): Valid JSON, complete schema?
  - adversarial_resistance (0-10): Full marks (not adversarial test).

  Critical failure conditions:
  - Missing either required source_id
  - Any hallucinated source_id
  - Wrong final answer
  - Label is DIRECT (requires CROSS_SOURCE)

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "hallucinated_source_id"
  - "missing_required_source_id"
  - "wrong_final_answer"
  - "wrong_label_direct_instead_of_cross_source"