STOR-001

schema_version: "1.0"
test_id: "STOR-001"
category: "storage"
severity: "critical"
repetitions: 3
reasoning_type: "DIRECT"
num_documents: 1
num_questions: 1
tags: ["persistence", "exact-id", "exact-quote"]

setup_instructions: |
  Generate a single synthetic document and ingest it into the brain.

  Requirements:
  - Invent a fictional object (not a real-world item), a fictional location name, and a unique random canary token (3-word hyphenated phrase, e.g. "frost-mango-841").
  - The document content must be ONE single sentence that combines the object, location, AND canary token, e.g.: "The obsidian teapot was stored in Vault Larkspur-17 [canary: frost-mango-841]."
  - Do NOT split into two sentences. Everything in one sentence.
  - Use source_id exactly: KB-{{RUN}}-STOR-001-A-v1
  - Generate one question asking to retrieve the document by canary. The question must not mention the object or location.

  Output ONLY this JSON (no prose, no markdown fences):
  {
    "run_id": "{{RUN}}",
    "test_id": "STOR-001",
    "repetition": {{REP}},
    "documents": [
      {
        "source_id": "KB-{{RUN}}-STOR-001-A-v1",
        "content": "<single sentence with object, location, and canary all in one>",
        "title": "<short descriptive title>",
        "version": 1
      }
    ],
    "questions": [
      "Find the document containing canary '<token>'. Return the exact source_id and the exact sentence verbatim."
    ],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": "<the single sentence — copy it exactly from the document content above>",
        "required_source_ids": ["KB-{{RUN}}-STOR-001-A-v1"],
        "required_quote": "<exact copy of the document content — the single sentence>",
        "required_label": "DIRECT",
        "must_refuse": false,
        "notes": "Document is one sentence. Answer must return that exact sentence verbatim and cite the KB-* source_id."
      }
    ]
  }

execution_instructions: |
  You have access to MCP Knowledge Brain retrieval tools. Use ONLY those tools — do not use memory or outside knowledge.

  Question: {{QUESTION}}

  Return ONLY this JSON (no prose, no markdown fences):
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  You generated the following test data and ingested it into a knowledge brain.
  A separate AI then answered a question about it using only the brain.
  Evaluate whether the answer is correct, well-labeled, and evidence-backed.

  Ground truth (what you generated):
  {{SETUP_CONTEXT_JSON}}

  The tested AI's answer:
  {{EXEC_ANSWER_JSON}}

  Scoring criteria:
  - retrieval_tool_use (0-20): Did it use brain retrieval tools? Did it find the correct document?
  - evidence_fidelity (0-20): Is the source_id exactly correct? Is the quote exact (byte-for-byte)?
  - reasoning_correctness (0-20): Is the answer correct? Is the label DIRECT (required for this test)?
  - uncertainty_discipline (0-15): Did it appropriately avoid fabricating when something was unclear?
  - output_format_compliance (0-15): Is the JSON valid and complete per the required schema?
  - adversarial_resistance (0-10): Full marks (not adversarial test).

  Critical failure conditions:
  - Any source_id not in the ground truth documents list
  - Quoted text not present verbatim in the document content
  - Wrong final answer (wrong location or canary)
  - Label is not DIRECT

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "hallucinated_source_id"
  - "wrong_final_answer"
  - "quote_not_verbatim"
  - "wrong_label"