← All Test Cases

STOR-003

storage

high

Repetitions

Documents

Questions

Reasoning

DIRECT

persistence distractor no-contamination

📖 In Plain English

What this category tests

Does the brain remember exactly what was ingested?

How the test works

A document with a unique canary token is ingested. The test then asks for that document back and checks the source ID and quote are byte-for-byte identical.

Why it matters

The most basic brain function — if storage is broken, nothing else matters.

⚙️ How a single rep runs

① Generate

Model creates 2 synthetic documents and 1 question with unique canary tokens

→ Fresh content per run prevents memorization and proves real retrieval

② Ingest (MCP)

Model calls brain_ingest to store the 2 documents

→ Tests the brain's storage and indexing pipeline

③ Query (MCP)

Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)

→ Core test — does the brain return correct evidence and let the model build a faithful answer?

④ Evaluate

Model judges the answer against ground truth (the document it generated in phase 1)

→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 3 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1359 chars

Generate two synthetic documents about two similar objects pointing to different locations.
Ingest both. Generate a question about only ONE of the objects.

Requirements:
- Document A: "<adjective-A> compass points to <location-A>." Use e.g. "silver" and "Pier Seven".
- Document B: "<adjective-B> compass points to <location-B>." Use a different adjective and location (e.g. "brass" and "Pier Nine").
- Both use "compass" as object type but different adjectives.
- source_ids: KB-{{RUN}}-STOR-003-A-v1 and KB-{{RUN}}-STOR-003-B-v1
- The question asks where the adjective-A compass points. The answer must come from A only.

Output ONLY this JSON:
{
  "run_id": "{{RUN}}",
  "test_id": "STOR-003",
  "repetition": {{REP}},
  "documents": [
    {"source_id": "KB-{{RUN}}-STOR-003-A-v1", "content": "<Doc A>", "title": "<title A>", "version": 1},
    {"source_id": "KB-{{RUN}}-STOR-003-B-v1", "content": "<Doc B>", "title": "<title B>", "version": 1}
  ],
  "questions": ["Where does the <adjective-A> compass point?"],
  "expected_answers": [{
    "question_index": 0,
    "correct_answer": "<location-A>",
    "required_source_ids": ["KB-{{RUN}}-STOR-003-A-v1"],
    "required_quote": "<exact Doc A sentence>",
    "required_label": "DIRECT",
    "must_refuse": false,
    "notes": "Answer must use only Doc A. No contamination from Doc B's location."
  }]
}

▶ ② Execution Instructions 119 chars

Use ONLY MCP Knowledge Brain retrieval tools.
Question: {{QUESTION}}
Return ONLY this JSON:
{{REQUIRED_ANSWER_SCHEMA}}

⚖ ③ Judge Instructions 349 chars

Ground truth:
{{SETUP_CONTEXT_JSON}}

Tested AI answer:
{{EXEC_ANSWER_JSON}}

Check: correct location from Doc A, NOT Doc B's location, correct source_id (A not B), DIRECT label.
Critical failures: answered with Doc B's location, hallucinated source_id, cited both sources for a single-source answer.

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

hallucinated_source_id
wrong_final_answer_used_distractor_doc
cited_wrong_source

Recent Run History

2 runs

When	Run ID	Pass Rate	Avg Score	Reps
2026-05-24 13:08	20260524T130808Z-kqze	0%	55.0	0/1	View →
2026-05-24 11:37	20260524T113756Z-kduj	0%	73.0	0/1	View →

📄 Raw YAML cases/storage/STOR-003.yaml

schema_version: "1.0"
test_id: "STOR-003"
category: "storage"
severity: "high"
repetitions: 3
reasoning_type: "DIRECT"
num_documents: 2
num_questions: 1
tags: ["persistence", "distractor", "no-contamination"]

setup_instructions: |
  Generate two synthetic documents about two similar objects pointing to different locations.
  Ingest both. Generate a question about only ONE of the objects.

  Requirements:
  - Document A: "<adjective-A> compass points to <location-A>." Use e.g. "silver" and "Pier Seven".
  - Document B: "<adjective-B> compass points to <location-B>." Use a different adjective and location (e.g. "brass" and "Pier Nine").
  - Both use "compass" as object type but different adjectives.
  - source_ids: KB-{{RUN}}-STOR-003-A-v1 and KB-{{RUN}}-STOR-003-B-v1
  - The question asks where the adjective-A compass points. The answer must come from A only.

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "STOR-003",
    "repetition": {{REP}},
    "documents": [
      {"source_id": "KB-{{RUN}}-STOR-003-A-v1", "content": "<Doc A>", "title": "<title A>", "version": 1},
      {"source_id": "KB-{{RUN}}-STOR-003-B-v1", "content": "<Doc B>", "title": "<title B>", "version": 1}
    ],
    "questions": ["Where does the <adjective-A> compass point?"],
    "expected_answers": [{
      "question_index": 0,
      "correct_answer": "<location-A>",
      "required_source_ids": ["KB-{{RUN}}-STOR-003-A-v1"],
      "required_quote": "<exact Doc A sentence>",
      "required_label": "DIRECT",
      "must_refuse": false,
      "notes": "Answer must use only Doc A. No contamination from Doc B's location."
    }]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools.
  Question: {{QUESTION}}
  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  Check: correct location from Doc A, NOT Doc B's location, correct source_id (A not B), DIRECT label.
  Critical failures: answered with Doc B's location, hallucinated source_id, cited both sources for a single-source answer.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "hallucinated_source_id"
  - "wrong_final_answer_used_distractor_doc"
  - "cited_wrong_source"