DDP-001

schema_version: "1.0"
test_id: "DDP-001"
category: "deduplication"
severity: "high"
repetitions: 3
reasoning_type: "DIRECT"
num_documents: 1
num_questions: 1
tags: ["dedup", "reingest", "duplicate-detection", "data-quality"]

setup_instructions: |
  Generate one synthetic document. It will be ingested in Phase 2.
  Phase 3 will attempt to ingest the SAME document again to test duplicate detection.

  Requirements:
  - Invent a unique fictional fact (1-2 sentences) with a canary token.
  - source_id: KB-{{RUN}}-DDP-001-A-v1
  - The question instructs the model to re-ingest and report the dedup result.

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "DDP-001",
    "repetition": {{REP}},
    "documents": [
      {
        "source_id": "KB-{{RUN}}-DDP-001-A-v1",
        "content": "<1-2 sentences with a unique fictional fact and canary token>",
        "title": "<short title>",
        "version": 1
      }
    ],
    "questions": [
      "Call brain_ingest again with EXACTLY the same document that was already ingested: source_id=KB-{{RUN}}-DDP-001-A-v1, same content as in setup_context. Report whether the brain returned duplicate_detected=true and what source_id it returned. Then search for the canary token and confirm only ONE source exists."
    ],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": "duplicate_detected: true",
        "required_source_ids": ["KB-{{RUN}}-DDP-001-A-v1"],
        "required_label": "DIRECT",
        "must_refuse": false,
        "notes": "The brain must detect that the same content was already ingested. The second ingest call should return duplicate_detected=true (or equivalent). Only one source should exist after two ingest calls."
      }
    ]
  }

execution_instructions: |
  You have access to MCP Knowledge Brain tools including brain_ingest.

  Step 1: Get the document details from setup_context (source_id and content).
  Step 2: Call brain_ingest again with EXACTLY the same source_id and content that was already ingested in Phase 2.
  Step 3: Examine the response — did it return duplicate_detected=true or indicate the source already exists?
  Step 4: Call brain_search with the canary token to confirm only one source exists.

  Question: {{QUESTION}}

  Return ONLY this JSON (answer should be "duplicate_detected: true" if detected, "duplicate_not_detected" if not):
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  The AI was asked to ingest the same document a second time and check for duplicate detection.
  The brain should have returned duplicate_detected=true (or indicated the source already exists).

  Scoring:
  - retrieval_tool_use (0-20): Did it call brain_ingest a second time with the same content? Did it also search to verify?
  - evidence_fidelity (0-20): Did it correctly identify the existing source_id? Is the source_id KB-{{RUN}}-DDP-001-A-v1?
  - reasoning_correctness (0-20): Did the brain detect the duplicate? Is answer "duplicate_detected: true"?
  - uncertainty_discipline (0-15): Did it report the brain's actual response accurately?
  - output_format_compliance (0-15): Valid JSON?
  - adversarial_resistance (0-10): Full marks.

  Critical failures: did not call brain_ingest a second time, misreported the dedup response, reported two separate sources exist.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "did_not_reingest"
  - "duplicate_not_detected_by_brain"
  - "two_sources_created"