← All Test Cases

PHR-005

exact phrase

critical

Repetitions

Documents

Questions

Reasoning

DIRECT

exact-quote embedded-quotes final-word

📖 In Plain English

What this category tests

Does the brain quote source text verbatim, with no paraphrasing?

How the test works

Documents contain very specific phrases. The test asks for those phrases back and checks every character — including spaces, punctuation, and special characters — matches exactly.

Why it matters

Paraphrased quotes break legal/audit use cases where exact wording matters.

⚙️ How a single rep runs

① Generate

Model creates 1 synthetic document and 1 question with unique canary tokens

→ Fresh content per run prevents memorization and proves real retrieval

② Ingest (MCP)

Model calls brain_ingest to store the 1 document

→ Tests the brain's storage and indexing pipeline

③ Query (MCP)

Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)

→ Core test — does the brain return correct evidence and let the model build a faithful answer?

④ Evaluate

Model judges the answer against ground truth (the document it generated in phase 1)

→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 3 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1233 chars

Generate a document with a sentence where the final word is wrapped in double quotes. Ingest it.
Ask to return the exact sentence and source_id.

Requirements:
- Content: 'The final word in the note was "<fictional-word>".'
- Choose a unique fictional word (e.g. "horizon", "celadon", "verity").
- source_id: KB-{{RUN}}-PHR-005-A-v1
- Question: "Return the exact sentence and source_id from the document about the final word."

Output ONLY this JSON:
{
  "run_id": "{{RUN}}",
  "test_id": "PHR-005",
  "repetition": {{REP}},
  "documents": [
    {"source_id": "KB-{{RUN}}-PHR-005-A-v1", "content": "The final word in the note was \"<fictional-word>\".", "title": "<title>", "version": 1}
  ],
  "questions": ["Return the exact sentence and source_id from the document about the final word in the note."],
  "expected_answers": [{
    "question_index": 0,
    "correct_answer": "The final word in the note was \"<fictional-word>\".",
    "required_source_ids": ["KB-{{RUN}}-PHR-005-A-v1"],
    "required_quote": "The final word in the note was \"<fictional-word>\".",
    "required_label": "DIRECT",
    "must_refuse": false,
    "notes": "Quotes around the final word must be preserved. Missing quotes is a fidelity failure."
  }]
}

▶ ② Execution Instructions 119 chars

Use ONLY MCP Knowledge Brain retrieval tools.
Question: {{QUESTION}}
Return ONLY this JSON:
{{REQUIRED_ANSWER_SCHEMA}}

⚖ ③ Judge Instructions 325 chars

Ground truth:
{{SETUP_CONTEXT_JSON}}

Tested AI answer:
{{EXEC_ANSWER_JSON}}

Check: the final word must be wrapped in double quotes in the returned sentence. Missing quotes is a failure.
Critical failures: missing quotes around final word, wrong word, hallucinated source_id.

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

missing_quotes_around_final_word
quote_paraphrased
hallucinated_source_id

Recent Run History

3 runs

When	Run ID	Pass Rate	Avg Score	Reps
2026-05-24 13:08	20260524T130808Z-kqze	0%	10.0	0/1	View →
2026-05-24 12:41	20260524T124148Z-z2do	100%	100.0	1/1	View →
2026-05-24 11:37	20260524T113756Z-kduj	100%	100.0	1/1	View →

📄 Raw YAML cases/exact_phrase/PHR-005.yaml

schema_version: "1.0"
test_id: "PHR-005"
category: "exact_phrase"
severity: "critical"
repetitions: 3
reasoning_type: "DIRECT"
num_documents: 1
num_questions: 1
tags: ["exact-quote", "embedded-quotes", "final-word"]

setup_instructions: |
  Generate a document with a sentence where the final word is wrapped in double quotes. Ingest it.
  Ask to return the exact sentence and source_id.

  Requirements:
  - Content: 'The final word in the note was "<fictional-word>".'
  - Choose a unique fictional word (e.g. "horizon", "celadon", "verity").
  - source_id: KB-{{RUN}}-PHR-005-A-v1
  - Question: "Return the exact sentence and source_id from the document about the final word."

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "PHR-005",
    "repetition": {{REP}},
    "documents": [
      {"source_id": "KB-{{RUN}}-PHR-005-A-v1", "content": "The final word in the note was \"<fictional-word>\".", "title": "<title>", "version": 1}
    ],
    "questions": ["Return the exact sentence and source_id from the document about the final word in the note."],
    "expected_answers": [{
      "question_index": 0,
      "correct_answer": "The final word in the note was \"<fictional-word>\".",
      "required_source_ids": ["KB-{{RUN}}-PHR-005-A-v1"],
      "required_quote": "The final word in the note was \"<fictional-word>\".",
      "required_label": "DIRECT",
      "must_refuse": false,
      "notes": "Quotes around the final word must be preserved. Missing quotes is a fidelity failure."
    }]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools.
  Question: {{QUESTION}}
  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  Check: the final word must be wrapped in double quotes in the returned sentence. Missing quotes is a failure.
  Critical failures: missing quotes around final word, wrong word, hallucinated source_id.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "missing_quotes_around_final_word"
  - "quote_paraphrased"
  - "hallucinated_source_id"