← All Test Cases

SID-005

exact source id

critical

Repetitions

Documents

Questions

Reasoning

DIRECT

source-id trap text-vs-metadata

📖 In Plain English

What this category tests

Does the brain return the exact KB-* source label, not an internal ID?

How the test works

After ingesting documents, the test asks the brain to cite specific sources. Citations must use the KB-* label format the document was stored with, never the internal src_* IDs.

Why it matters

Source IDs are the brain's contract for provenance. If they're wrong, evidence can't be traced.

⚙️ How a single rep runs

① Generate

Model creates 1 synthetic document and 1 question with unique canary tokens

→ Fresh content per run prevents memorization and proves real retrieval

② Ingest (MCP)

Model calls brain_ingest to store the 1 document

→ Tests the brain's storage and indexing pipeline

③ Query (MCP)

Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)

→ Core test — does the brain return correct evidence and let the model build a faithful answer?

④ Evaluate

Model judges the answer against ground truth (the document it generated in phase 1)

→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 3 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1304 chars

Generate a document whose text contains a phrase that looks like a source_id but is NOT the real source_id.
The real source_id is metadata. Ask for the actual source_id.

Requirements:
- Document content must contain a phrase like "source-id trap" or a fake ID-like string embedded in the text (e.g. "The phrase SRC-FAKE-999 appears here, but the real identifier is in the metadata.").
- The actual source_id is KB-{{RUN}}-SID-005-A-v1 (metadata, not in the text).
- Question: "What is the exact source_id of the document containing the phrase 'source-id trap'?"

Output ONLY this JSON:
{
  "run_id": "{{RUN}}",
  "test_id": "SID-005",
  "repetition": {{REP}},
  "documents": [
    {"source_id": "KB-{{RUN}}-SID-005-A-v1", "content": "<content containing fake ID-like phrase>", "title": "<title>", "version": 1}
  ],
  "questions": ["What is the exact source_id of the document containing the phrase that looks like a fake source identifier?"],
  "expected_answers": [{
    "question_index": 0,
    "correct_answer": "KB-{{RUN}}-SID-005-A-v1",
    "required_source_ids": ["KB-{{RUN}}-SID-005-A-v1"],
    "required_quote": null,
    "required_label": "DIRECT",
    "must_refuse": false,
    "notes": "Must return the real metadata source_id, NOT the fake ID string found inside the document text."
  }]
}

▶ ② Execution Instructions 119 chars

Use ONLY MCP Knowledge Brain retrieval tools.
Question: {{QUESTION}}
Return ONLY this JSON:
{{REQUIRED_ANSWER_SCHEMA}}

⚖ ③ Judge Instructions 398 chars

Ground truth:
{{SETUP_CONTEXT_JSON}}

Tested AI answer:
{{EXEC_ANSWER_JSON}}

Check: the returned source_id must be the real metadata ID "KB-{{RUN}}-SID-005-A-v1".
If the agent returned a fake ID-like string found inside the document text, that is a critical failure.
Critical failures: returned in-text fake ID as source_id, hallucinated source_id.

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

returned_in_text_phrase_as_source_id
hallucinated_source_id

Recent Run History

3 runs

When	Run ID	Pass Rate	Avg Score	Reps
2026-05-24 13:08	20260524T130808Z-kqze	0%	20.0	0/1	View →
2026-05-24 12:41	20260524T124148Z-z2do	0%	65.0	0/1	View →
2026-05-24 11:37	20260524T113756Z-kduj	0%	82.0	0/1	View →

📄 Raw YAML cases/exact_source_id/SID-005.yaml

schema_version: "1.0"
test_id: "SID-005"
category: "exact_source_id"
severity: "critical"
repetitions: 3
reasoning_type: "DIRECT"
num_documents: 1
num_questions: 1
tags: ["source-id", "trap", "text-vs-metadata"]

setup_instructions: |
  Generate a document whose text contains a phrase that looks like a source_id but is NOT the real source_id.
  The real source_id is metadata. Ask for the actual source_id.

  Requirements:
  - Document content must contain a phrase like "source-id trap" or a fake ID-like string embedded in the text (e.g. "The phrase SRC-FAKE-999 appears here, but the real identifier is in the metadata.").
  - The actual source_id is KB-{{RUN}}-SID-005-A-v1 (metadata, not in the text).
  - Question: "What is the exact source_id of the document containing the phrase 'source-id trap'?"

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "SID-005",
    "repetition": {{REP}},
    "documents": [
      {"source_id": "KB-{{RUN}}-SID-005-A-v1", "content": "<content containing fake ID-like phrase>", "title": "<title>", "version": 1}
    ],
    "questions": ["What is the exact source_id of the document containing the phrase that looks like a fake source identifier?"],
    "expected_answers": [{
      "question_index": 0,
      "correct_answer": "KB-{{RUN}}-SID-005-A-v1",
      "required_source_ids": ["KB-{{RUN}}-SID-005-A-v1"],
      "required_quote": null,
      "required_label": "DIRECT",
      "must_refuse": false,
      "notes": "Must return the real metadata source_id, NOT the fake ID string found inside the document text."
    }]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools.
  Question: {{QUESTION}}
  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  Check: the returned source_id must be the real metadata ID "KB-{{RUN}}-SID-005-A-v1".
  If the agent returned a fake ID-like string found inside the document text, that is a critical failure.
  Critical failures: returned in-text fake ID as source_id, hallucinated source_id.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "returned_in_text_phrase_as_source_id"
  - "hallucinated_source_id"