← All Test Cases

PHR-001

exact phrase

critical

Repetitions

Documents

Questions

Reasoning

DIRECT

exact-quote no-paraphrase verbatim

📖 In Plain English

What this category tests

Does the brain quote source text verbatim, with no paraphrasing?

How the test works

Documents contain very specific phrases. The test asks for those phrases back and checks every character — including spaces, punctuation, and special characters — matches exactly.

Why it matters

Paraphrased quotes break legal/audit use cases where exact wording matters.

⚙️ How a single rep runs

① Generate

Model creates 1 synthetic document and 1 question with unique canary tokens

→ Fresh content per run prevents memorization and proves real retrieval

② Ingest (MCP)

Model calls brain_ingest to store the 1 document

→ Tests the brain's storage and indexing pipeline

③ Query (MCP)

Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)

→ Core test — does the brain return correct evidence and let the model build a faithful answer?

④ Evaluate

Model judges the answer against ground truth (the document it generated in phase 1)

→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 3 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1147 chars

Generate a document with one sentence containing an unusual verb (e.g. "flickered", "shuddered", "glimmered").
Ingest it. Ask to return the exact sentence containing that verb.

Requirements:
- One sentence with an unusual verb describing a fictional object's behavior. Example: "The lantern flickered exactly three times before dawn."
- source_id: KB-{{RUN}}-PHR-001-A-v1
- Question: "Return the exact sentence from the brain containing the word '<unusual-verb>'."

Output ONLY this JSON:
{
  "run_id": "{{RUN}}",
  "test_id": "PHR-001",
  "repetition": {{REP}},
  "documents": [
    {"source_id": "KB-{{RUN}}-PHR-001-A-v1", "content": "<sentence>", "title": "<title>", "version": 1}
  ],
  "questions": ["Return the exact sentence from the brain containing the word '<unusual-verb>'."],
  "expected_answers": [{
    "question_index": 0,
    "correct_answer": "<exact sentence>",
    "required_source_ids": ["KB-{{RUN}}-PHR-001-A-v1"],
    "required_quote": "<exact sentence>",
    "required_label": "DIRECT",
    "must_refuse": false,
    "notes": "Quote must be verbatim — no paraphrasing, no casing changes, no added or removed words."
  }]
}

▶ ② Execution Instructions 119 chars

Use ONLY MCP Knowledge Brain retrieval tools.
Question: {{QUESTION}}
Return ONLY this JSON:
{{REQUIRED_ANSWER_SCHEMA}}

⚖ ③ Judge Instructions 364 chars

Ground truth:
{{SETUP_CONTEXT_JSON}}

Tested AI answer:
{{EXEC_ANSWER_JSON}}

Check: the evidence_quotes in claims must contain the exact sentence verbatim. Any paraphrase, word substitution, or casing change is a failure.
Critical failures: paraphrased quote, changed casing, missing words, hallucinated source_id.

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

quote_paraphrased
quote_casing_changed
hallucinated_source_id

Recent Run History

3 runs

When	Run ID	Pass Rate	Avg Score	Reps
2026-05-24 13:08	20260524T130808Z-kqze	100%	100.0	1/1	View →
2026-05-24 12:41	20260524T124148Z-z2do	100%	100.0	1/1	View →
2026-05-24 11:37	20260524T113756Z-kduj	100%	100.0	1/1	View →

📄 Raw YAML cases/exact_phrase/PHR-001.yaml

schema_version: "1.0"
test_id: "PHR-001"
category: "exact_phrase"
severity: "critical"
repetitions: 3
reasoning_type: "DIRECT"
num_documents: 1
num_questions: 1
tags: ["exact-quote", "no-paraphrase", "verbatim"]

setup_instructions: |
  Generate a document with one sentence containing an unusual verb (e.g. "flickered", "shuddered", "glimmered").
  Ingest it. Ask to return the exact sentence containing that verb.

  Requirements:
  - One sentence with an unusual verb describing a fictional object's behavior. Example: "The lantern flickered exactly three times before dawn."
  - source_id: KB-{{RUN}}-PHR-001-A-v1
  - Question: "Return the exact sentence from the brain containing the word '<unusual-verb>'."

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "PHR-001",
    "repetition": {{REP}},
    "documents": [
      {"source_id": "KB-{{RUN}}-PHR-001-A-v1", "content": "<sentence>", "title": "<title>", "version": 1}
    ],
    "questions": ["Return the exact sentence from the brain containing the word '<unusual-verb>'."],
    "expected_answers": [{
      "question_index": 0,
      "correct_answer": "<exact sentence>",
      "required_source_ids": ["KB-{{RUN}}-PHR-001-A-v1"],
      "required_quote": "<exact sentence>",
      "required_label": "DIRECT",
      "must_refuse": false,
      "notes": "Quote must be verbatim — no paraphrasing, no casing changes, no added or removed words."
    }]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools.
  Question: {{QUESTION}}
  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  Check: the evidence_quotes in claims must contain the exact sentence verbatim. Any paraphrase, word substitution, or casing change is a failure.
  Critical failures: paraphrased quote, changed casing, missing words, hallucinated source_id.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "quote_paraphrased"
  - "quote_casing_changed"
  - "hallucinated_source_id"