IDN-003

schema_version: "1.0"
test_id: "IDN-003"
category: "identifier"
severity: "high"
repetitions: 3
reasoning_type: "DIRECT"
num_documents: 1
num_questions: 1
tags: ["structured-id", "pattern-preservation", "alphanumeric", "exact-token"]

setup_instructions: |
  Generate one document containing a structured alphanumeric identifier (like "INR-7823" or "REF-4421-X")
  and verify the brain preserves and retrieves it exactly.

  Requirements:
  - Invent a fictional structured identifier in the format: 3 uppercase letters, hyphen, 4 digits
    (e.g. "TRK-9182", "INV-3341", "REF-5509"). Call this IDENTIFIER.
  - Document: ONE sentence stating that IDENTIFIER is assigned to a fictional entity.
    Example: "Identifier TRK-9182 is assigned to shipment Aurora-Delta in warehouse zone 7."
  - source_id: KB-{{RUN}}-IDN-003-A-v1
  - Question: search for the IDENTIFIER exactly and return the sentence.

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "IDN-003",
    "repetition": {{REP}},
    "documents": [
      {
        "source_id": "KB-{{RUN}}-IDN-003-A-v1",
        "content": "<single sentence with the structured identifier>",
        "title": "<short title>",
        "version": 1
      }
    ],
    "questions": [
      "Search the brain for identifier '<IDENTIFIER>' exactly as written. Return the source_id and verbatim sentence."
    ],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": "<the document sentence — exact copy>",
        "required_source_ids": ["KB-{{RUN}}-IDN-003-A-v1"],
        "required_quote": "<exact copy of document content>",
        "required_label": "DIRECT",
        "must_refuse": false,
        "notes": "The structured identifier pattern (e.g. TRK-9182) must be preserved exactly and be searchable. Any mutation of the pattern (e.g. trk9182, TRK 9182) is a critical failure."
      }
    ]
  }

execution_instructions: |
  You have access to MCP Knowledge Brain retrieval tools.
  Use brain_search with the exact identifier string from the question.

  Question: {{QUESTION}}

  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  The AI must retrieve a document using a structured alphanumeric identifier (e.g. TRK-9182).
  This tests that the brain preserves hyphenated alphanumeric patterns exactly and makes them searchable.

  Scoring:
  - retrieval_tool_use (0-20): Did it search with the exact identifier string?
  - evidence_fidelity (0-20): Correct source_id? Exact verbatim quote including the identifier pattern?
  - reasoning_correctness (0-20): Correct answer sentence returned?
  - uncertainty_discipline (0-15): No invented content?
  - output_format_compliance (0-15): Valid JSON?
  - adversarial_resistance (0-10): Full marks.

  Critical failures: wrong/hallucinated source_id, identifier pattern mutated in the quote, document not found.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "hallucinated_source_id"
  - "identifier_pattern_mutated"
  - "document_not_found"