← All Test Cases

NUM-002

numerical

high
Repetitions
3
Documents
3
Questions
1
Reasoning
CROSS_SOURCE
numerical-aggregation sum-across-docs cross-source

📖 In Plain English

What this category tests

Can the brain compare and aggregate numbers across documents?

How the test works

Documents contain quantitative facts. The test asks for comparisons ('which is largest?', 'what's the total?') requiring numeric reasoning across multiple sources.

Why it matters

Many real questions involve numbers — figures, dates, counts, percentages.

⚙️ How a single rep runs

① Generate
Model creates 3 synthetic documents and 1 question with unique canary tokens
→ Fresh content per run prevents memorization and proves real retrieval
② Ingest (MCP)
Model calls brain_ingest to store the 3 documents
→ Tests the brain's storage and indexing pipeline
③ Query (MCP)
Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)
→ Core test — does the brain return correct evidence and let the model build a faithful answer?
④ Evaluate
Model judges the answer against ground truth (the document it generated in phase 1)
→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 3 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1454 chars 
Generate three documents each stating a quantity. Ask for the total sum.

Requirements:
- Doc A: "Department <A> has <N-A> items." (e.g., 12)
- Doc B: "Department <B> has <N-B> items." (e.g., 8)
- Doc C: "Department <C> has <N-C> items." (e.g., 25)
- Use fictional department names and item types.
- source_ids: KB-{{RUN}}-NUM-002-A-v1, KB-{{RUN}}-NUM-002-B-v1, KB-{{RUN}}-NUM-002-C-v1.
- Question: "How many <items> are there in total across all three departments?"

Output ONLY this JSON:
{
  "run_id": "{{RUN}}",
  "test_id": "NUM-002",
  "repetition": {{REP}},
  "documents": [
    {"source_id": "KB-{{RUN}}-NUM-002-A-v1", "content": "Department <A> has <N-A> <items>.", "title": "<title>", "version": 1},
    {"source_id": "KB-{{RUN}}-NUM-002-B-v1", "content": "Department <B> has <N-B> <items>.", "title": "<title>", "version": 1},
    {"source_id": "KB-{{RUN}}-NUM-002-C-v1", "content": "Department <C> has <N-C> <items>.", "title": "<title>", "version": 1}
  ],
  "questions": ["How many <items> are there in total across all three departments?"],
  "expected_answers": [{
    "question_index": 0,
    "correct_answer": "<N-A + N-B + N-C>",
    "required_source_ids": ["KB-{{RUN}}-NUM-002-A-v1", "KB-{{RUN}}-NUM-002-B-v1", "KB-{{RUN}}-NUM-002-C-v1"],
    "required_quote": null,
    "required_label": "CROSS_SOURCE",
    "must_refuse": false,
    "notes": "Must retrieve all 3 values and sum correctly. Any arithmetic error is a failure."
  }]
}
▶ ② Execution Instructions 144 chars 
Use ONLY MCP Knowledge Brain retrieval tools with domain="brain-test".
Question: {{QUESTION}}
Return ONLY this JSON:
{{REQUIRED_ANSWER_SCHEMA}}
⚖ ③ Judge Instructions 283 chars 
Ground truth:
{{SETUP_CONTEXT_JSON}}

Tested AI answer:
{{EXEC_ANSWER_JSON}}

Check: all 3 source_ids cited, correct sum computed, CROSS_SOURCE label.
Critical failures: wrong sum, missing source, hallucinated source_id, DIRECT label.

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

Recent Run History

3 runs
When Run ID Pass Rate Avg Score Reps
2026-05-24 13:08 20260524T130808Z-kqze 100% 100.0 1/1 View →
2026-05-24 12:41 20260524T124148Z-z2do 100% 100.0 1/1 View →
2026-05-24 11:37 20260524T113756Z-kduj 100% 100.0 1/1 View →
📄 Raw YAML cases/numerical/NUM-002.yaml 
schema_version: "1.0"
test_id: "NUM-002"
category: "numerical"
severity: "high"
repetitions: 3
reasoning_type: "CROSS_SOURCE"
num_documents: 3
num_questions: 1
tags: ["numerical-aggregation", "sum-across-docs", "cross-source"]

setup_instructions: |
  Generate three documents each stating a quantity. Ask for the total sum.

  Requirements:
  - Doc A: "Department <A> has <N-A> items." (e.g., 12)
  - Doc B: "Department <B> has <N-B> items." (e.g., 8)
  - Doc C: "Department <C> has <N-C> items." (e.g., 25)
  - Use fictional department names and item types.
  - source_ids: KB-{{RUN}}-NUM-002-A-v1, KB-{{RUN}}-NUM-002-B-v1, KB-{{RUN}}-NUM-002-C-v1.
  - Question: "How many <items> are there in total across all three departments?"

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "NUM-002",
    "repetition": {{REP}},
    "documents": [
      {"source_id": "KB-{{RUN}}-NUM-002-A-v1", "content": "Department <A> has <N-A> <items>.", "title": "<title>", "version": 1},
      {"source_id": "KB-{{RUN}}-NUM-002-B-v1", "content": "Department <B> has <N-B> <items>.", "title": "<title>", "version": 1},
      {"source_id": "KB-{{RUN}}-NUM-002-C-v1", "content": "Department <C> has <N-C> <items>.", "title": "<title>", "version": 1}
    ],
    "questions": ["How many <items> are there in total across all three departments?"],
    "expected_answers": [{
      "question_index": 0,
      "correct_answer": "<N-A + N-B + N-C>",
      "required_source_ids": ["KB-{{RUN}}-NUM-002-A-v1", "KB-{{RUN}}-NUM-002-B-v1", "KB-{{RUN}}-NUM-002-C-v1"],
      "required_quote": null,
      "required_label": "CROSS_SOURCE",
      "must_refuse": false,
      "notes": "Must retrieve all 3 values and sum correctly. Any arithmetic error is a failure."
    }]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools with domain="brain-test".
  Question: {{QUESTION}}
  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  Check: all 3 source_ids cited, correct sum computed, CROSS_SOURCE label.
  Critical failures: wrong sum, missing source, hallucinated source_id, DIRECT label.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "wrong_sum"
  - "missing_required_source_id"
  - "hallucinated_source_id"
  - "wrong_label_not_cross_source"