← All Test Cases

SDR-001

semantic drift

high

Repetitions

Documents

Questions

Reasoning

DIRECT

semantic-search paraphrase-query synonym-retrieval

📖 In Plain English

What this category tests

Does the brain find documents when the query uses synonyms?

How the test works

A document uses one term ('authorized'); the query uses a synonym ('approved'). The brain must bridge the vocabulary gap and retrieve the document.

Why it matters

Users don't always know the exact phrasing in the brain. Synonym handling is essential for usability.

⚙️ How a single rep runs

① Generate

Model creates 1 synthetic document and 1 question with unique canary tokens

→ Fresh content per run prevents memorization and proves real retrieval

② Ingest (MCP)

Model calls brain_ingest to store the 1 document

→ Tests the brain's storage and indexing pipeline

③ Query (MCP)

Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)

→ Core test — does the brain return correct evidence and let the model build a faithful answer?

④ Evaluate

Model judges the answer against ground truth (the document it generated in phase 1)

→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 5 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1382 chars

Generate one document stating where a fictional object is stored, using specific vocabulary.
Then generate a paraphrased question that uses different but semantically equivalent words.

Requirements:
- Document uses one set of words: e.g. "The vessel is kept in Chamber 7."
  ("vessel" = container, "kept" = stored, "Chamber" = room/vault)
- Generate a question using synonyms that don't appear in the document.
  e.g. "Where is the container stored?" or "In which room is the receptacle housed?"
- The paraphrased question MUST NOT use any word from the document content verbatim.
- source_id: KB-{{RUN}}-SDR-001-A-v1

Output ONLY this JSON:
{
  "run_id": "{{RUN}}",
  "test_id": "SDR-001",
  "repetition": {{REP}},
  "documents": [
    {"source_id": "KB-{{RUN}}-SDR-001-A-v1", "content": "<sentence using vocabulary set A>", "title": "<title>", "version": 1}
  ],
  "questions": ["<paraphrased question using vocabulary set B — no overlap with document words>"],
  "expected_answers": [{
    "question_index": 0,
    "correct_answer": "<the location from the document>",
    "required_source_ids": ["KB-{{RUN}}-SDR-001-A-v1"],
    "required_quote": null,
    "required_label": "DIRECT",
    "must_refuse": false,
    "notes": "The brain must retrieve the document via semantic search despite no keyword overlap. Failure means semantic search is not working or too narrow."
  }]
}

▶ ② Execution Instructions 314 chars

Use ONLY MCP Knowledge Brain retrieval tools (brain_search with semantic search enabled).
Question: {{QUESTION}}
The answer may require semantic/conceptual matching — use brain_search and look for semantically related content even if keywords don't match exactly.
Return ONLY this JSON:
{{REQUIRED_ANSWER_SCHEMA}}

⚖ ③ Judge Instructions 514 chars

Ground truth:
{{SETUP_CONTEXT_JSON}}

Tested AI answer:
{{EXEC_ANSWER_JSON}}

This test uses a paraphrased query with no keyword overlap with the document.
The brain must find the document via semantic search. Correct answer = the location, DIRECT label, correct source_id.
If the brain failed to retrieve the document: this is a semantic search failure, not a reasoning failure.
Critical failures: wrong location, did not retrieve document, hallucinated source_id.

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

semantic_search_failed_to_retrieve_document
wrong_final_answer
hallucinated_source_id

Recent Run History

2 runs

When	Run ID	Pass Rate	Avg Score	Reps
2026-05-24 13:08	20260524T130808Z-kqze	0%	37.0	0/1	View →
2026-05-24 11:37	20260524T113756Z-kduj	100%	100.0	1/1	View →

📄 Raw YAML cases/semantic_drift/SDR-001.yaml

schema_version: "1.0"
test_id: "SDR-001"
category: "semantic_drift"
severity: "high"
repetitions: 5
reasoning_type: "DIRECT"
num_documents: 1
num_questions: 1
tags: ["semantic-search", "paraphrase-query", "synonym-retrieval"]

setup_instructions: |
  Generate one document stating where a fictional object is stored, using specific vocabulary.
  Then generate a paraphrased question that uses different but semantically equivalent words.

  Requirements:
  - Document uses one set of words: e.g. "The vessel is kept in Chamber 7."
    ("vessel" = container, "kept" = stored, "Chamber" = room/vault)
  - Generate a question using synonyms that don't appear in the document.
    e.g. "Where is the container stored?" or "In which room is the receptacle housed?"
  - The paraphrased question MUST NOT use any word from the document content verbatim.
  - source_id: KB-{{RUN}}-SDR-001-A-v1

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "SDR-001",
    "repetition": {{REP}},
    "documents": [
      {"source_id": "KB-{{RUN}}-SDR-001-A-v1", "content": "<sentence using vocabulary set A>", "title": "<title>", "version": 1}
    ],
    "questions": ["<paraphrased question using vocabulary set B — no overlap with document words>"],
    "expected_answers": [{
      "question_index": 0,
      "correct_answer": "<the location from the document>",
      "required_source_ids": ["KB-{{RUN}}-SDR-001-A-v1"],
      "required_quote": null,
      "required_label": "DIRECT",
      "must_refuse": false,
      "notes": "The brain must retrieve the document via semantic search despite no keyword overlap. Failure means semantic search is not working or too narrow."
    }]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools (brain_search with semantic search enabled).
  Question: {{QUESTION}}
  The answer may require semantic/conceptual matching — use brain_search and look for semantically related content even if keywords don't match exactly.
  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  This test uses a paraphrased query with no keyword overlap with the document.
  The brain must find the document via semantic search. Correct answer = the location, DIRECT label, correct source_id.
  If the brain failed to retrieve the document: this is a semantic search failure, not a reasoning failure.
  Critical failures: wrong location, did not retrieve document, hallucinated source_id.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "semantic_search_failed_to_retrieve_document"
  - "wrong_final_answer"
  - "hallucinated_source_id"