DDP-002

schema_version: "1.0"
test_id: "DDP-002"
category: "deduplication"
severity: "medium"
repetitions: 3
reasoning_type: "DIRECT"
num_documents: 1
num_questions: 1
tags: ["dedup", "near-duplicate", "content-hash", "data-quality"]

setup_instructions: |
  Generate one synthetic document. Phase 3 will attempt to ingest a NEAR-DUPLICATE
  (same content, different source_id) to test content-hash deduplication.

  Requirements:
  - Invent a unique fictional fact (2-3 sentences) with a canary token.
  - source_id for original: KB-{{RUN}}-DDP-002-A-v1
  - The question instructs the model to ingest the same content under a NEW source_id.

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "DDP-002",
    "repetition": {{REP}},
    "documents": [
      {
        "source_id": "KB-{{RUN}}-DDP-002-A-v1",
        "content": "<2-3 sentences with a unique fictional fact and canary token>",
        "title": "<short title>",
        "version": 1
      }
    ],
    "questions": [
      "Call brain_ingest with the SAME content as KB-{{RUN}}-DDP-002-A-v1 but using a DIFFERENT source_id: KB-{{RUN}}-DDP-002-B-v1. Report whether the brain detected a duplicate or created a new source. Then search for the canary and report how many sources were found."
    ],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": "duplicate_detected: true",
        "required_source_ids": ["KB-{{RUN}}-DDP-002-A-v1"],
        "required_label": "DIRECT",
        "must_refuse": false,
        "notes": "The brain should detect that the same content body was already stored, even under a different source_id label. If content-hash dedup works, only 1 source should exist and duplicate_detected=true."
      }
    ]
  }

execution_instructions: |
  You have access to MCP Knowledge Brain tools including brain_ingest.

  Step 1: Get the document content from setup_context.
  Step 2: Call brain_ingest with the SAME content but source_id=KB-{{RUN}}-DDP-002-B-v1 (different label, same text).
  Step 3: Examine the response — duplicate_detected=true means content-hash dedup is working.
  Step 4: Search for the canary token and count how many distinct sources appear.

  Question: {{QUESTION}}

  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  The AI ingested the same content under a different source label to test content-hash deduplication.
  If the brain has content-hash dedup, it should return duplicate_detected=true even for a different source_id.

  Scoring:
  - retrieval_tool_use (0-20): Did it call brain_ingest with a different source_id but same content? Did it search after?
  - evidence_fidelity (0-20): Correct report of what the brain returned?
  - reasoning_correctness (0-20): Did the brain detect the duplicate? If not, was only one source found in search?
  - uncertainty_discipline (0-15): Accurate reporting of brain responses?
  - output_format_compliance (0-15): Valid JSON?
  - adversarial_resistance (0-10): Full marks.

  Note: If the brain does NOT detect content-hash duplicates (only label duplicates), this test will fail — that is a valid finding about the brain's dedup capability.

  Critical failures: did not attempt reingest with different label, misreported number of sources found.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "did_not_reingest_with_different_label"
  - "misreported_source_count"