SUP-001

schema_version: "1.0"
test_id: "SUP-001"
category: "supersession"
severity: "high"
repetitions: 2
reasoning_type: "DIRECT"
num_documents: 2
num_questions: 1
tags: ["auto-supersession", "confidence", "conflict-resolution", "data-quality"]

setup_instructions: |
  Generate two CONFLICTING documents about the same entity.
  The first has low confidence (0.5), the second has high confidence (1.0).
  Auto-supersession should resolve the conflict in favour of the high-confidence claim,
  so brain_pending_reviews should NOT show an open conflict between them.

  Requirements:
  - Invent a fictional entity and ONE fact about it (e.g. "The Vega Module power output is 400W")
  - Doc 1: low-confidence version (claim A, confidence=0.5)
  - Doc 2: high-confidence conflicting version (claim B, confidence=1.0)
  - They must be genuinely conflicting (different values for same attribute)
  - source_ids: KB-{{RUN}}-SUP-001-A-v1 and KB-{{RUN}}-SUP-001-B-v1
  - Question: check if the low-confidence claim was superseded

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "SUP-001",
    "repetition": {{REP}},
    "documents": [
      {
        "source_id": "KB-{{RUN}}-SUP-001-A-v1",
        "content": "<claim A — lower value, canary1>",
        "title": "<title>",
        "version": 1
      },
      {
        "source_id": "KB-{{RUN}}-SUP-001-B-v1",
        "content": "<claim B — higher/different value, canary2>",
        "title": "<title>",
        "version": 1
      }
    ],
    "questions": [
      "After ingesting both claims (A at confidence=0.5, B at confidence=1.0): check brain_pending_reviews for type=conflict. The A-B conflict should NOT appear (auto-superseded). Also search for the entity — only claim B should be active."
    ],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": "auto-superseded: no open conflict, only claim B active",
        "required_source_ids": ["KB-{{RUN}}-SUP-001-B-v1"],
        "required_label": "DIRECT",
        "must_refuse": false,
        "notes": "High confidence (1.0) claim should auto-supersede low confidence (0.5) claim. brain_pending_reviews should not show A-B as an open conflict."
      }
    ]
  }

execution_instructions: |
  You have access to MCP Knowledge Brain tools.

  ══════════════════════════════════════════════════════════════
  STEP 1: Fetch the actual test documents by label
  ══════════════════════════════════════════════════════════════
  Before doing anything else, fetch both documents:
    brain_fetch("brain://source/by-label/KB-{{RUN}}-SUP-001-{{REP}}-A-v1")
    brain_fetch("brain://source/by-label/KB-{{RUN}}-SUP-001-{{REP}}-B-v1")

  Save the content of both as DOC_A and DOC_B.
  Use ONLY DOC_A and DOC_B to understand what the claims are — never use memory.

  STEP 2: Ingest using the content from DOC_A and DOC_B
  Ingest DOC_A with brain_ingest using extracted.claims[0].confidence from setup_context.
  Ingest DOC_B with brain_ingest using extracted.claims[0].confidence from setup_context.

  STEP 3: Check brain_pending_reviews with type="conflict"
  STEP 4: Report whether the A-B conflict appears or was auto-superseded.

  Answer only using what you fetched and observed — not from memory.

  Question: {{QUESTION}}

  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}


judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  Claim A (confidence=0.5) and Claim B (confidence=1.0) are conflicting.
  Auto-supersession should have resolved this without creating an open conflict.

  Scoring:
  - retrieval_tool_use (0-20): Used brain_ingest with correct confidences + brain_pending_reviews?
  - evidence_fidelity (0-20): Correct source_ids reported? Accurately reported pending reviews result?
  - reasoning_correctness (0-20): Correctly identified whether conflict was auto-superseded?
  - uncertainty_discipline (0-15): Accurate reporting of brain state?
  - output_format_compliance (0-15): Valid JSON?
  - adversarial_resistance (0-10): Full marks.

  Critical failures: did not check brain_pending_reviews, misreported auto-supersession result.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "did_not_check_pending_reviews"
  - "conflict_still_open_when_should_be_superseded"