← All Test Cases

NUM-001

numerical

high

Repetitions

Documents

Questions

Reasoning

CROSS_SOURCE

numerical-comparison cross-source greater-than

📖 In Plain English

What this category tests

Can the brain compare and aggregate numbers across documents?

How the test works

Documents contain quantitative facts. The test asks for comparisons ('which is largest?', 'what's the total?') requiring numeric reasoning across multiple sources.

Why it matters

Many real questions involve numbers — figures, dates, counts, percentages.

⚙️ How a single rep runs

① Generate

Model creates 2 synthetic documents and 1 question with unique canary tokens

→ Fresh content per run prevents memorization and proves real retrieval

② Ingest (MCP)

Model calls brain_ingest to store the 2 documents

→ Tests the brain's storage and indexing pipeline

③ Query (MCP)

Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)

→ Core test — does the brain return correct evidence and let the model build a faithful answer?

④ Evaluate

Model judges the answer against ground truth (the document it generated in phase 1)

→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 3 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1343 chars

Generate two documents each stating a numerical value for the same type of thing.
Ask which value is higher/lower.

Requirements:
- Doc A: "<Policy-A> has a limit of <Number-A>." (e.g., 150)
- Doc B: "<Policy-B> has a limit of <Number-B>." (e.g., 300)
- Number-A and Number-B must be clearly different integers.
- source_ids: KB-{{RUN}}-NUM-001-A-v1 and KB-{{RUN}}-NUM-001-B-v1.
- Question: "Which policy has the higher limit, <Policy-A> or <Policy-B>? By how much?"

Output ONLY this JSON:
{
  "run_id": "{{RUN}}",
  "test_id": "NUM-001",
  "repetition": {{REP}},
  "documents": [
    {"source_id": "KB-{{RUN}}-NUM-001-A-v1", "content": "<Policy-A> has a limit of <Number-A>.", "title": "<title>", "version": 1},
    {"source_id": "KB-{{RUN}}-NUM-001-B-v1", "content": "<Policy-B> has a limit of <Number-B>.", "title": "<title>", "version": 1}
  ],
  "questions": ["Which policy has the higher limit, <Policy-A> or <Policy-B>? By how much?"],
  "expected_answers": [{
    "question_index": 0,
    "correct_answer": "<higher-policy> by <difference>",
    "required_source_ids": ["KB-{{RUN}}-NUM-001-A-v1", "KB-{{RUN}}-NUM-001-B-v1"],
    "required_quote": null,
    "required_label": "CROSS_SOURCE",
    "must_refuse": false,
    "notes": "Must retrieve both values, compare correctly, and state the difference. CROSS_SOURCE required."
  }]
}

▶ ② Execution Instructions 144 chars

Use ONLY MCP Knowledge Brain retrieval tools with domain="brain-test".
Question: {{QUESTION}}
Return ONLY this JSON:
{{REQUIRED_ANSWER_SCHEMA}}

⚖ ③ Judge Instructions 338 chars

Ground truth:
{{SETUP_CONTEXT_JSON}}

Tested AI answer:
{{EXEC_ANSWER_JSON}}

Check: correct policy named as higher, correct numerical difference, both source_ids cited, CROSS_SOURCE label.
Critical failures: wrong policy named, wrong difference, missing source_id, hallucinated source_id.

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

wrong_policy_named_as_higher
wrong_numerical_difference
missing_required_source_id
hallucinated_source_id

Recent Run History

2 runs

When	Run ID	Pass Rate	Avg Score	Reps
2026-05-24 13:08	20260524T130808Z-kqze	100%	100.0	1/1	View →
2026-05-24 11:37	20260524T113756Z-kduj	100%	100.0	1/1	View →

📄 Raw YAML cases/numerical/NUM-001.yaml

schema_version: "1.0"
test_id: "NUM-001"
category: "numerical"
severity: "high"
repetitions: 3
reasoning_type: "CROSS_SOURCE"
num_documents: 2
num_questions: 1
tags: ["numerical-comparison", "cross-source", "greater-than"]

setup_instructions: |
  Generate two documents each stating a numerical value for the same type of thing.
  Ask which value is higher/lower.

  Requirements:
  - Doc A: "<Policy-A> has a limit of <Number-A>." (e.g., 150)
  - Doc B: "<Policy-B> has a limit of <Number-B>." (e.g., 300)
  - Number-A and Number-B must be clearly different integers.
  - source_ids: KB-{{RUN}}-NUM-001-A-v1 and KB-{{RUN}}-NUM-001-B-v1.
  - Question: "Which policy has the higher limit, <Policy-A> or <Policy-B>? By how much?"

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "NUM-001",
    "repetition": {{REP}},
    "documents": [
      {"source_id": "KB-{{RUN}}-NUM-001-A-v1", "content": "<Policy-A> has a limit of <Number-A>.", "title": "<title>", "version": 1},
      {"source_id": "KB-{{RUN}}-NUM-001-B-v1", "content": "<Policy-B> has a limit of <Number-B>.", "title": "<title>", "version": 1}
    ],
    "questions": ["Which policy has the higher limit, <Policy-A> or <Policy-B>? By how much?"],
    "expected_answers": [{
      "question_index": 0,
      "correct_answer": "<higher-policy> by <difference>",
      "required_source_ids": ["KB-{{RUN}}-NUM-001-A-v1", "KB-{{RUN}}-NUM-001-B-v1"],
      "required_quote": null,
      "required_label": "CROSS_SOURCE",
      "must_refuse": false,
      "notes": "Must retrieve both values, compare correctly, and state the difference. CROSS_SOURCE required."
    }]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools with domain="brain-test".
  Question: {{QUESTION}}
  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  Check: correct policy named as higher, correct numerical difference, both source_ids cited, CROSS_SOURCE label.
  Critical failures: wrong policy named, wrong difference, missing source_id, hallucinated source_id.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "wrong_policy_named_as_higher"
  - "wrong_numerical_difference"
  - "missing_required_source_id"
  - "hallucinated_source_id"