← All Test Cases

PHR-003

exact phrase

high

Repetitions

Documents

Questions

Reasoning

DIRECT

exact-quote unicode em-dash

📖 In Plain English

What this category tests

Does the brain quote source text verbatim, with no paraphrasing?

How the test works

Documents contain very specific phrases. The test asks for those phrases back and checks every character — including spaces, punctuation, and special characters — matches exactly.

Why it matters

Paraphrased quotes break legal/audit use cases where exact wording matters.

⚙️ How a single rep runs

① Generate

Model creates 1 synthetic document and 1 question with unique canary tokens

→ Fresh content per run prevents memorization and proves real retrieval

② Ingest (MCP)

Model calls brain_ingest to store the 1 document

→ Tests the brain's storage and indexing pipeline

③ Query (MCP)

Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)

→ Core test — does the brain return correct evidence and let the model build a faithful answer?

④ Evaluate

Model judges the answer against ground truth (the document it generated in phase 1)

→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 3 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1323 chars

Generate a document containing a label with an em dash (—) contrasted with a hyphen (-). Ingest it.
Ask to return the exact label sentence.

Requirements:
- Content: "The label read: <WORD-A>—<WORD-B>, not <WORD-A>-<WORD-B>." (em dash vs hyphen)
- Example: "The label read: ALPHA—OMEGA, not ALPHA-OMEGA."
- Use fictional ALL-CAPS word pair. The em dash (—, U+2014) must be in the document.
- source_id: KB-{{RUN}}-PHR-003-A-v1
- Question: "Return the exact label sentence from the brain document."

Output ONLY this JSON:
{
  "run_id": "{{RUN}}",
  "test_id": "PHR-003",
  "repetition": {{REP}},
  "documents": [
    {"source_id": "KB-{{RUN}}-PHR-003-A-v1", "content": "The label read: <WORD-A>—<WORD-B>, not <WORD-A>-<WORD-B>.", "title": "<title>", "version": 1}
  ],
  "questions": ["Return the exact label sentence from the brain document, preserving all punctuation."],
  "expected_answers": [{
    "question_index": 0,
    "correct_answer": "The label read: <WORD-A>—<WORD-B>, not <WORD-A>-<WORD-B>.",
    "required_source_ids": ["KB-{{RUN}}-PHR-003-A-v1"],
    "required_quote": "The label read: <WORD-A>—<WORD-B>, not <WORD-A>-<WORD-B>.",
    "required_label": "DIRECT",
    "must_refuse": false,
    "notes": "The em dash (—) must be preserved. Substituting a hyphen (-) or double-hyphen (--) is a failure."
  }]
}

▶ ② Execution Instructions 119 chars

Use ONLY MCP Knowledge Brain retrieval tools.
Question: {{QUESTION}}
Return ONLY this JSON:
{{REQUIRED_ANSWER_SCHEMA}}

⚖ ③ Judge Instructions 341 chars

Ground truth:
{{SETUP_CONTEXT_JSON}}

Tested AI answer:
{{EXEC_ANSWER_JSON}}

Check: the em dash character (—, U+2014) must be present in the returned quote. A hyphen or double-hyphen in its place is a fidelity failure.
Critical failures: em dash replaced with hyphen, hallucinated source_id.

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

em_dash_replaced_with_hyphen
unicode_character_lost
hallucinated_source_id

Recent Run History

3 runs

When	Run ID	Pass Rate	Avg Score	Reps
2026-05-24 13:08	20260524T130808Z-kqze	0%	31.0	0/1	View →
2026-05-24 12:41	20260524T124148Z-z2do	100%	100.0	1/1	View →
2026-05-24 11:37	20260524T113756Z-kduj	100%	100.0	1/1	View →

📄 Raw YAML cases/exact_phrase/PHR-003.yaml

schema_version: "1.0"
test_id: "PHR-003"
category: "exact_phrase"
severity: "high"
repetitions: 3
reasoning_type: "DIRECT"
num_documents: 1
num_questions: 1
tags: ["exact-quote", "unicode", "em-dash"]

setup_instructions: |
  Generate a document containing a label with an em dash (—) contrasted with a hyphen (-). Ingest it.
  Ask to return the exact label sentence.

  Requirements:
  - Content: "The label read: <WORD-A>—<WORD-B>, not <WORD-A>-<WORD-B>." (em dash vs hyphen)
  - Example: "The label read: ALPHA—OMEGA, not ALPHA-OMEGA."
  - Use fictional ALL-CAPS word pair. The em dash (—, U+2014) must be in the document.
  - source_id: KB-{{RUN}}-PHR-003-A-v1
  - Question: "Return the exact label sentence from the brain document."

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "PHR-003",
    "repetition": {{REP}},
    "documents": [
      {"source_id": "KB-{{RUN}}-PHR-003-A-v1", "content": "The label read: <WORD-A>—<WORD-B>, not <WORD-A>-<WORD-B>.", "title": "<title>", "version": 1}
    ],
    "questions": ["Return the exact label sentence from the brain document, preserving all punctuation."],
    "expected_answers": [{
      "question_index": 0,
      "correct_answer": "The label read: <WORD-A>—<WORD-B>, not <WORD-A>-<WORD-B>.",
      "required_source_ids": ["KB-{{RUN}}-PHR-003-A-v1"],
      "required_quote": "The label read: <WORD-A>—<WORD-B>, not <WORD-A>-<WORD-B>.",
      "required_label": "DIRECT",
      "must_refuse": false,
      "notes": "The em dash (—) must be preserved. Substituting a hyphen (-) or double-hyphen (--) is a failure."
    }]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools.
  Question: {{QUESTION}}
  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  Check: the em dash character (—, U+2014) must be present in the returned quote. A hyphen or double-hyphen in its place is a fidelity failure.
  Critical failures: em dash replaced with hyphen, hallucinated source_id.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "em_dash_replaced_with_hyphen"
  - "unicode_character_lost"
  - "hallucinated_source_id"