← All Test Cases

PHR-002

exact phrase

critical

Repetitions

Documents

Questions

Reasoning

DIRECT

exact-quote punctuation speech-marks

📖 In Plain English

What this category tests

Does the brain quote source text verbatim, with no paraphrasing?

How the test works

Documents contain very specific phrases. The test asks for those phrases back and checks every character — including spaces, punctuation, and special characters — matches exactly.

Why it matters

Paraphrased quotes break legal/audit use cases where exact wording matters.

⚙️ How a single rep runs

① Generate

Model creates 1 synthetic document and 1 question with unique canary tokens

→ Fresh content per run prevents memorization and proves real retrieval

② Ingest (MCP)

Model calls brain_ingest to store the 1 document

→ Tests the brain's storage and indexing pipeline

③ Query (MCP)

Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)

→ Core test — does the brain return correct evidence and let the model build a faithful answer?

④ Evaluate

Model judges the answer against ground truth (the document it generated in phase 1)

→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 3 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1222 chars

Generate a document containing a direct quote (speech) from a fictional person. Ingest it.
Ask to return exactly what the person said, including quotation marks.

Requirements:
- Content: '<Person> said, "<imperative sentence not to do something>."'
- Example: 'Milo said, "Do not move the green vase."'
- Use a fictional person name and a unique imperative sentence.
- source_id: KB-{{RUN}}-PHR-002-A-v1
- Question: "Quote exactly what <Person> said according to the brain document."

Output ONLY this JSON:
{
  "run_id": "{{RUN}}",
  "test_id": "PHR-002",
  "repetition": {{REP}},
  "documents": [
    {"source_id": "KB-{{RUN}}-PHR-002-A-v1", "content": "<Person> said, \"<imperative sentence>.\"", "title": "<title>", "version": 1}
  ],
  "questions": ["Quote exactly what <Person> said according to the brain document."],
  "expected_answers": [{
    "question_index": 0,
    "correct_answer": "\"<imperative sentence>.\"",
    "required_source_ids": ["KB-{{RUN}}-PHR-002-A-v1"],
    "required_quote": "<Person> said, \"<imperative sentence>.\"",
    "required_label": "DIRECT",
    "must_refuse": false,
    "notes": "The quotation marks must be present in the answer. Missing open/close quotes is a failure."
  }]
}

▶ ② Execution Instructions 119 chars

Use ONLY MCP Knowledge Brain retrieval tools.
Question: {{QUESTION}}
Return ONLY this JSON:
{{REQUIRED_ANSWER_SCHEMA}}

⚖ ③ Judge Instructions 331 chars

Ground truth:
{{SETUP_CONTEXT_JSON}}

Tested AI answer:
{{EXEC_ANSWER_JSON}}

Check: the answer must include the exact spoken text with its quotation marks. Losing the quotes is a fidelity failure.
Critical failures: missing quotation marks, changed wording, hallucinated source_id.

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

missing_quotation_marks
quote_paraphrased
hallucinated_source_id

Recent Run History

3 runs

When	Run ID	Pass Rate	Avg Score	Reps
2026-05-24 13:08	20260524T130808Z-kqze	100%	95.0	1/1	View →
2026-05-24 12:41	20260524T124148Z-z2do	100%	100.0	1/1	View →
2026-05-24 11:37	20260524T113756Z-kduj	100%	100.0	1/1	View →

📄 Raw YAML cases/exact_phrase/PHR-002.yaml

schema_version: "1.0"
test_id: "PHR-002"
category: "exact_phrase"
severity: "critical"
repetitions: 3
reasoning_type: "DIRECT"
num_documents: 1
num_questions: 1
tags: ["exact-quote", "punctuation", "speech-marks"]

setup_instructions: |
  Generate a document containing a direct quote (speech) from a fictional person. Ingest it.
  Ask to return exactly what the person said, including quotation marks.

  Requirements:
  - Content: '<Person> said, "<imperative sentence not to do something>."'
  - Example: 'Milo said, "Do not move the green vase."'
  - Use a fictional person name and a unique imperative sentence.
  - source_id: KB-{{RUN}}-PHR-002-A-v1
  - Question: "Quote exactly what <Person> said according to the brain document."

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "PHR-002",
    "repetition": {{REP}},
    "documents": [
      {"source_id": "KB-{{RUN}}-PHR-002-A-v1", "content": "<Person> said, \"<imperative sentence>.\"", "title": "<title>", "version": 1}
    ],
    "questions": ["Quote exactly what <Person> said according to the brain document."],
    "expected_answers": [{
      "question_index": 0,
      "correct_answer": "\"<imperative sentence>.\"",
      "required_source_ids": ["KB-{{RUN}}-PHR-002-A-v1"],
      "required_quote": "<Person> said, \"<imperative sentence>.\"",
      "required_label": "DIRECT",
      "must_refuse": false,
      "notes": "The quotation marks must be present in the answer. Missing open/close quotes is a failure."
    }]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools.
  Question: {{QUESTION}}
  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  Check: the answer must include the exact spoken text with its quotation marks. Losing the quotes is a fidelity failure.
  Critical failures: missing quotation marks, changed wording, hallucinated source_id.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "missing_quotation_marks"
  - "quote_paraphrased"
  - "hallucinated_source_id"