← All Test Cases

RAW-005

raw text

high

Repetitions

Documents

Questions

Reasoning

DIRECT

raw-text braces json-like

📖 In Plain English

What this category tests

Are special characters preserved through ingest and retrieval?

How the test works

Documents contain tabs, Unicode, JSON snippets, literal newlines, and other tricky text. The test checks every character round-trips intact.

Why it matters

Knowledge often contains structured fragments (code, JSON, formulas) that must survive the brain's normalization.

⚙️ How a single rep runs

① Generate

Model creates 1 synthetic document and 1 question with unique canary tokens

→ Fresh content per run prevents memorization and proves real retrieval

② Ingest (MCP)

Model calls brain_ingest to store the 1 document

→ Tests the brain's storage and indexing pipeline

③ Query (MCP)

Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)

→ Core test — does the brain return correct evidence and let the model build a faithful answer?

④ Evaluate

Model judges the answer against ground truth (the document it generated in phase 1)

→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 3 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1223 chars

Generate a document with JSON-like syntax (curly braces, colons, quotes). Ingest it.
Ask to return the raw text exactly.

Requirements:
- Content: a short JSON-like string (not actual JSON, just looks like it). Example: 'JSON-ish: {"k": "v", "n": 3}'
- Use fictional key names.
- source_id: KB-{{RUN}}-RAW-005-A-v1
- Question: "Return the exact raw text of document KB-{{RUN}}-RAW-005-A-v1, preserving all braces and quotes."

Output ONLY this JSON:
{
  "run_id": "{{RUN}}",
  "test_id": "RAW-005",
  "repetition": {{REP}},
  "documents": [
    {"source_id": "KB-{{RUN}}-RAW-005-A-v1", "content": "JSON-ish: {\"<key1>\": \"<val1>\", \"<key2>\": <num>}", "title": "<title>", "version": 1}
  ],
  "questions": ["Return the exact raw text of document KB-{{RUN}}-RAW-005-A-v1, preserving all braces, quotes, and colons."],
  "expected_answers": [{
    "question_index": 0,
    "correct_answer": "JSON-ish: {\"<key1>\": \"<val1>\", \"<key2>\": <num>}",
    "required_source_ids": ["KB-{{RUN}}-RAW-005-A-v1"],
    "required_quote": "JSON-ish: {\"<key1>\": \"<val1>\", \"<key2>\": <num>}",
    "required_label": "DIRECT",
    "must_refuse": false,
    "notes": "Curly braces and internal quotes must be preserved exactly."
  }]
}

▶ ② Execution Instructions 119 chars

Use ONLY MCP Knowledge Brain retrieval tools.
Question: {{QUESTION}}
Return ONLY this JSON:
{{REQUIRED_ANSWER_SCHEMA}}

⚖ ③ Judge Instructions 377 chars

Ground truth:
{{SETUP_CONTEXT_JSON}}

Tested AI answer:
{{EXEC_ANSWER_JSON}}

Check: curly braces, quotes, colons in the content must all be preserved. Any escaping artifacts that alter the visible text are failures.
Critical failures: braces removed, quotes escaped differently changing visible content, hallucinated source_id.

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

braces_or_quotes_removed
content_altered_by_escaping
hallucinated_source_id

Recent Run History

2 runs

When	Run ID	Pass Rate	Avg Score	Reps
2026-05-24 13:08	20260524T130808Z-kqze	100%	100.0	1/1	View →
2026-05-24 11:37	20260524T113756Z-kduj	100%	100.0	1/1	View →

📄 Raw YAML cases/raw_text/RAW-005.yaml

schema_version: "1.0"
test_id: "RAW-005"
category: "raw_text"
severity: "high"
repetitions: 3
reasoning_type: "DIRECT"
num_documents: 1
num_questions: 1
tags: ["raw-text", "braces", "json-like"]

setup_instructions: |
  Generate a document with JSON-like syntax (curly braces, colons, quotes). Ingest it.
  Ask to return the raw text exactly.

  Requirements:
  - Content: a short JSON-like string (not actual JSON, just looks like it). Example: 'JSON-ish: {"k": "v", "n": 3}'
  - Use fictional key names.
  - source_id: KB-{{RUN}}-RAW-005-A-v1
  - Question: "Return the exact raw text of document KB-{{RUN}}-RAW-005-A-v1, preserving all braces and quotes."

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "RAW-005",
    "repetition": {{REP}},
    "documents": [
      {"source_id": "KB-{{RUN}}-RAW-005-A-v1", "content": "JSON-ish: {\"<key1>\": \"<val1>\", \"<key2>\": <num>}", "title": "<title>", "version": 1}
    ],
    "questions": ["Return the exact raw text of document KB-{{RUN}}-RAW-005-A-v1, preserving all braces, quotes, and colons."],
    "expected_answers": [{
      "question_index": 0,
      "correct_answer": "JSON-ish: {\"<key1>\": \"<val1>\", \"<key2>\": <num>}",
      "required_source_ids": ["KB-{{RUN}}-RAW-005-A-v1"],
      "required_quote": "JSON-ish: {\"<key1>\": \"<val1>\", \"<key2>\": <num>}",
      "required_label": "DIRECT",
      "must_refuse": false,
      "notes": "Curly braces and internal quotes must be preserved exactly."
    }]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools.
  Question: {{QUESTION}}
  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  Check: curly braces, quotes, colons in the content must all be preserved. Any escaping artifacts that alter the visible text are failures.
  Critical failures: braces removed, quotes escaped differently changing visible content, hallucinated source_id.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "braces_or_quotes_removed"
  - "content_altered_by_escaping"
  - "hallucinated_source_id"