SUP-002

schema_version: "1.0"
test_id: "SUP-002"
category: "supersession"
severity: "high"
repetitions: 2
reasoning_type: "UNCERTAIN"
num_documents: 2
num_questions: 1
tags: ["no-auto-supersession", "open-conflict", "similar-confidence", "data-quality"]

setup_instructions: |
  Generate two CONFLICTING documents with SIMILAR confidence levels (both ~0.7).
  Auto-supersession should NOT trigger — instead an open conflict should be created,
  visible in brain_pending_reviews.

  Requirements:
  - Invent a fictional entity and ONE fact about it
  - Doc 1: claim A at confidence=0.7
  - Doc 2: claim B (conflicting) at confidence=0.7
  - Both claims are equally plausible — brain should flag this as a conflict
  - source_ids: KB-{{RUN}}-SUP-002-A-v1 and KB-{{RUN}}-SUP-002-B-v1
  - Question: verify an open conflict exists in brain_pending_reviews

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "SUP-002",
    "repetition": {{REP}},
    "documents": [
      {
        "source_id": "KB-{{RUN}}-SUP-002-A-v1",
        "content": "<claim A about entity, canary1>",
        "title": "<title>",
        "version": 1
      },
      {
        "source_id": "KB-{{RUN}}-SUP-002-B-v1",
        "content": "<claim B — conflicting, canary2>",
        "title": "<title>",
        "version": 1
      }
    ],
    "questions": [
      "After ingesting both claims at confidence=0.7: check brain_pending_reviews for type=conflict. An open conflict between A and B SHOULD appear (not auto-superseded because confidences are equal)."
    ],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": "open conflict exists: both claims appear in pending reviews",
        "required_source_ids": ["KB-{{RUN}}-SUP-002-A-v1", "KB-{{RUN}}-SUP-002-B-v1"],
        "required_label": "UNCERTAIN",
        "must_refuse": false,
        "notes": "When both claims have equal confidence, auto-supersession does NOT trigger. An open conflict should appear in brain_pending_reviews. Label UNCERTAIN because the true value is ambiguous."
      }
    ]
  }

execution_instructions: |
  You have access to MCP Knowledge Brain tools.

  ══════════════════════════════════════════════════════════════
  STEP 1: Fetch the actual test documents by label
  ══════════════════════════════════════════════════════════════
  Before doing anything else, fetch both documents:
    brain_fetch("brain://source/by-label/KB-{{RUN}}-SUP-002-{{REP}}-A-v1")
    brain_fetch("brain://source/by-label/KB-{{RUN}}-SUP-002-{{REP}}-B-v1")

  Save the content of both as DOC_A and DOC_B.
  Use ONLY DOC_A and DOC_B to understand what the claims are — never use memory.

  STEP 2: Ingest using the content from DOC_A and DOC_B
  Ingest DOC_A with brain_ingest using extracted.claims[0].confidence from setup_context.
  Ingest DOC_B with brain_ingest using extracted.claims[0].confidence from setup_context.

  STEP 3: Check brain_pending_reviews with type="conflict"
  STEP 4: Report whether the A-B conflict appears or was auto-superseded.

  Answer only using what you fetched and observed — not from memory.

  Question: {{QUESTION}}

  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}


judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  Both claims have equal confidence (0.7). Auto-supersession should NOT have triggered.
  An open conflict should be visible in brain_pending_reviews.

  Scoring:
  - retrieval_tool_use (0-20): Ingested with equal confidences + checked pending_reviews?
  - evidence_fidelity (0-20): Accurately reported what pending_reviews returned?
  - reasoning_correctness (0-20): Correctly identified open conflict (not auto-superseded)?
  - uncertainty_discipline (0-15): Used UNCERTAIN label (genuinely ambiguous situation)?
  - output_format_compliance (0-15): Valid JSON?
  - adversarial_resistance (0-10): Full marks.

  Critical failures: reported auto-superseded when conflict should be open, did not check pending_reviews.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "did_not_check_pending_reviews"
  - "conflict_incorrectly_auto_superseded"