← All Test Cases

PHR-004

exact phrase

high
Repetitions
3
Documents
1
Questions
1
Reasoning
DIRECT
exact-quote slash-spacing special-chars

📖 In Plain English

What this category tests

Does the brain quote source text verbatim, with no paraphrasing?

How the test works

Documents contain very specific phrases. The test asks for those phrases back and checks every character — including spaces, punctuation, and special characters — matches exactly.

Why it matters

Paraphrased quotes break legal/audit use cases where exact wording matters.

⚙️ How a single rep runs

① Generate
Model creates 1 synthetic document and 1 question with unique canary tokens
→ Fresh content per run prevents memorization and proves real retrieval
② Ingest (MCP)
Model calls brain_ingest to store the 1 document
→ Tests the brain's storage and indexing pipeline
③ Query (MCP)
Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)
→ Core test — does the brain return correct evidence and let the model build a faithful answer?
④ Evaluate
Model judges the answer against ground truth (the document it generated in phase 1)
→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 3 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1176 chars 
Generate a document containing a hint with slash-separated words (with spaces around slashes). Ingest it.
Ask to return the exact hint line.

Requirements:
- Content: "Password hint: <word1> / <word2> / <word3>." (spaces around each slash)
- Choose three unrelated fictional words (e.g. "tulip / cedar / rain").
- source_id: KB-{{RUN}}-PHR-004-A-v1
- Question: "Return the exact password hint line from the brain document."

Output ONLY this JSON:
{
  "run_id": "{{RUN}}",
  "test_id": "PHR-004",
  "repetition": {{REP}},
  "documents": [
    {"source_id": "KB-{{RUN}}-PHR-004-A-v1", "content": "Password hint: <word1> / <word2> / <word3>.", "title": "<title>", "version": 1}
  ],
  "questions": ["Return the exact password hint line from the brain document."],
  "expected_answers": [{
    "question_index": 0,
    "correct_answer": "Password hint: <word1> / <word2> / <word3>.",
    "required_source_ids": ["KB-{{RUN}}-PHR-004-A-v1"],
    "required_quote": "Password hint: <word1> / <word2> / <word3>.",
    "required_label": "DIRECT",
    "must_refuse": false,
    "notes": "Spaces around slashes must be preserved. 'word1/word2/word3' without spaces is a failure."
  }]
}
▶ ② Execution Instructions 119 chars 
Use ONLY MCP Knowledge Brain retrieval tools.
Question: {{QUESTION}}
Return ONLY this JSON:
{{REQUIRED_ANSWER_SCHEMA}}
⚖ ③ Judge Instructions 327 chars 
Ground truth:
{{SETUP_CONTEXT_JSON}}

Tested AI answer:
{{EXEC_ANSWER_JSON}}

Check: spaces around the slashes must be preserved exactly. "word/word/word" is wrong; "word / word / word" is correct.
Critical failures: slash spacing removed, words changed, hallucinated source_id.

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

Recent Run History

3 runs
When Run ID Pass Rate Avg Score Reps
2026-05-24 13:08 20260524T130808Z-kqze 0% 73.0 0/1 View →
2026-05-24 12:41 20260524T124148Z-z2do 100% 100.0 1/1 View →
2026-05-24 11:37 20260524T113756Z-kduj 100% 100.0 1/1 View →
📄 Raw YAML cases/exact_phrase/PHR-004.yaml 
schema_version: "1.0"
test_id: "PHR-004"
category: "exact_phrase"
severity: "high"
repetitions: 3
reasoning_type: "DIRECT"
num_documents: 1
num_questions: 1
tags: ["exact-quote", "slash-spacing", "special-chars"]

setup_instructions: |
  Generate a document containing a hint with slash-separated words (with spaces around slashes). Ingest it.
  Ask to return the exact hint line.

  Requirements:
  - Content: "Password hint: <word1> / <word2> / <word3>." (spaces around each slash)
  - Choose three unrelated fictional words (e.g. "tulip / cedar / rain").
  - source_id: KB-{{RUN}}-PHR-004-A-v1
  - Question: "Return the exact password hint line from the brain document."

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "PHR-004",
    "repetition": {{REP}},
    "documents": [
      {"source_id": "KB-{{RUN}}-PHR-004-A-v1", "content": "Password hint: <word1> / <word2> / <word3>.", "title": "<title>", "version": 1}
    ],
    "questions": ["Return the exact password hint line from the brain document."],
    "expected_answers": [{
      "question_index": 0,
      "correct_answer": "Password hint: <word1> / <word2> / <word3>.",
      "required_source_ids": ["KB-{{RUN}}-PHR-004-A-v1"],
      "required_quote": "Password hint: <word1> / <word2> / <word3>.",
      "required_label": "DIRECT",
      "must_refuse": false,
      "notes": "Spaces around slashes must be preserved. 'word1/word2/word3' without spaces is a failure."
    }]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools.
  Question: {{QUESTION}}
  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  Check: spaces around the slashes must be preserved exactly. "word/word/word" is wrong; "word / word / word" is correct.
  Critical failures: slash spacing removed, words changed, hallucinated source_id.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "slash_spacing_removed"
  - "quote_paraphrased"
  - "hallucinated_source_id"