← All Test Cases

SDR-002

semantic drift

high

Repetitions

Documents

Questions

Reasoning

INFERRED

semantic-search action-synonym inferred-from-paraphrase

📖 In Plain English

What this category tests

Does the brain find documents when the query uses synonyms?

How the test works

A document uses one term ('authorized'); the query uses a synonym ('approved'). The brain must bridge the vocabulary gap and retrieve the document.

Why it matters

Users don't always know the exact phrasing in the brain. Synonym handling is essential for usability.

⚙️ How a single rep runs

① Generate

Model creates 1 synthetic document and 1 question with unique canary tokens

→ Fresh content per run prevents memorization and proves real retrieval

② Ingest (MCP)

Model calls brain_ingest to store the 1 document

→ Tests the brain's storage and indexing pipeline

③ Query (MCP)

Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)

→ Core test — does the brain return correct evidence and let the model build a faithful answer?

④ Evaluate

Model judges the answer against ground truth (the document it generated in phase 1)

→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 5 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1228 chars

Generate a document stating that a fictional person performed an approval action.
Generate a paraphrased question asking about a synonymous action.

Requirements:
- Document: "<Person> approved the <document/plan/design>." (e.g. "The engineer approved the blueprint.")
- Question uses synonyms: e.g. "Who signed off on the plan?" or "Who authorized the design?"
- The question must not use "approved" or the exact noun from the document.
- source_id: KB-{{RUN}}-SDR-002-A-v1

Output ONLY this JSON:
{
  "run_id": "{{RUN}}",
  "test_id": "SDR-002",
  "repetition": {{REP}},
  "documents": [
    {"source_id": "KB-{{RUN}}-SDR-002-A-v1", "content": "<Person> approved the <item>.", "title": "<title>", "version": 1}
  ],
  "questions": ["<paraphrased question about who authorized/signed-off — no keyword overlap>"],
  "expected_answers": [{
    "question_index": 0,
    "correct_answer": "<Person>",
    "required_source_ids": ["KB-{{RUN}}-SDR-002-A-v1"],
    "required_quote": null,
    "required_label": "INFERRED",
    "must_refuse": false,
    "notes": "INFERRED because 'signed off' and 'approved' are semantically equivalent but not identical. Brain must retrieve via semantic match. Failure = semantic search miss."
  }]
}

▶ ② Execution Instructions 148 chars

Use ONLY MCP Knowledge Brain retrieval tools with semantic search enabled.
Question: {{QUESTION}}
Return ONLY this JSON:
{{REQUIRED_ANSWER_SCHEMA}}

⚖ ③ Judge Instructions 426 chars

Ground truth:
{{SETUP_CONTEXT_JSON}}

Tested AI answer:
{{EXEC_ANSWER_JSON}}

The query uses action synonyms. The brain must retrieve via semantic match.
Correct: Person identified, INFERRED label (approved ≈ signed off), correct source_id.
Critical failures: did not find document, wrong person, DIRECT label (should be INFERRED for synonym reasoning), hallucinated source_id.

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

semantic_search_failed_to_retrieve_document
wrong_final_answer
hallucinated_source_id

Recent Run History

2 runs

When	Run ID	Pass Rate	Avg Score	Reps
2026-05-24 13:08	20260524T130808Z-kqze	0%	40.0	0/1	View →
2026-05-24 11:37	20260524T113756Z-kduj	100%	91.0	1/1	View →

📄 Raw YAML cases/semantic_drift/SDR-002.yaml

schema_version: "1.0"
test_id: "SDR-002"
category: "semantic_drift"
severity: "high"
repetitions: 5
reasoning_type: "INFERRED"
num_documents: 1
num_questions: 1
tags: ["semantic-search", "action-synonym", "inferred-from-paraphrase"]

setup_instructions: |
  Generate a document stating that a fictional person performed an approval action.
  Generate a paraphrased question asking about a synonymous action.

  Requirements:
  - Document: "<Person> approved the <document/plan/design>." (e.g. "The engineer approved the blueprint.")
  - Question uses synonyms: e.g. "Who signed off on the plan?" or "Who authorized the design?"
  - The question must not use "approved" or the exact noun from the document.
  - source_id: KB-{{RUN}}-SDR-002-A-v1

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "SDR-002",
    "repetition": {{REP}},
    "documents": [
      {"source_id": "KB-{{RUN}}-SDR-002-A-v1", "content": "<Person> approved the <item>.", "title": "<title>", "version": 1}
    ],
    "questions": ["<paraphrased question about who authorized/signed-off — no keyword overlap>"],
    "expected_answers": [{
      "question_index": 0,
      "correct_answer": "<Person>",
      "required_source_ids": ["KB-{{RUN}}-SDR-002-A-v1"],
      "required_quote": null,
      "required_label": "INFERRED",
      "must_refuse": false,
      "notes": "INFERRED because 'signed off' and 'approved' are semantically equivalent but not identical. Brain must retrieve via semantic match. Failure = semantic search miss."
    }]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools with semantic search enabled.
  Question: {{QUESTION}}
  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  The query uses action synonyms. The brain must retrieve via semantic match.
  Correct: Person identified, INFERRED label (approved ≈ signed off), correct source_id.
  Critical failures: did not find document, wrong person, DIRECT label (should be INFERRED for synonym reasoning), hallucinated source_id.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "semantic_search_failed_to_retrieve_document"
  - "wrong_final_answer"
  - "hallucinated_source_id"