← All Test Cases

ENT-003

entity disambiguation

critical

Repetitions

Documents

Questions

Reasoning

UNCERTAIN

entity-disambiguation ambiguous-reference pronoun-plus-entity

📖 In Plain English

What this category tests

Does the brain correctly distinguish entities with similar names?

How the test works

Documents describe different entities sharing a common word (e.g. Widget Pro / Widget Max / Widget Lite). The test asks about one specific entity and checks the others aren't returned by mistake.

Why it matters

Many domains have similar-named entities — confusing them produces wrong answers.

⚙️ How a single rep runs

① Generate

Model creates 2 synthetic documents and 1 question with unique canary tokens

→ Fresh content per run prevents memorization and proves real retrieval

② Ingest (MCP)

Model calls brain_ingest to store the 2 documents

→ Tests the brain's storage and indexing pipeline

③ Query (MCP)

Model answers the question using brain retrieval tools (search, fetch, context_pack, etc.)

→ Core test — does the brain return correct evidence and let the model build a faithful answer?

④ Evaluate

Model judges the answer against ground truth (the document it generated in phase 1)

→ Produces a score 0–100 with detailed sub-scores (retrieval, fidelity, reasoning, etc.)

This rep is run 3 times per test run. A pass requires score ≥ 85 and no critical failures.

🔬 Technical Instructions (raw prompts sent to AI)

🔧 ① Setup Instructions 1386 chars

Generate two documents about two people with the same job title but different names.
Ask a question using the job title (not the name) — the answer is ambiguous.

Requirements:
- Doc A: "<Name-A> is the <Title> of <Organization-A>."
- Doc B: "<Name-B> is the <Title> of <Organization-B>."
- Same job title, different names and organizations.
- source_ids: KB-{{RUN}}-ENT-003-A-v1 and KB-{{RUN}}-ENT-003-B-v1.
- Question: "Who is the <Title>?" — ambiguous because both are <Title> of different orgs.

Output ONLY this JSON:
{
  "run_id": "{{RUN}}",
  "test_id": "ENT-003",
  "repetition": {{REP}},
  "documents": [
    {"source_id": "KB-{{RUN}}-ENT-003-A-v1", "content": "<Name-A> is the <Title> of <Organization-A>.", "title": "<title>", "version": 1},
    {"source_id": "KB-{{RUN}}-ENT-003-B-v1", "content": "<Name-B> is the <Title> of <Organization-B>.", "title": "<title>", "version": 1}
  ],
  "questions": ["Who is the <Title>?"],
  "expected_answers": [{
    "question_index": 0,
    "correct_answer": null,
    "required_source_ids": ["KB-{{RUN}}-ENT-003-A-v1", "KB-{{RUN}}-ENT-003-B-v1"],
    "required_quote": null,
    "required_label": "UNCERTAIN",
    "must_refuse": false,
    "notes": "Two people hold the same title in different orgs. The question is ambiguous — must surface both and note the ambiguity. Picking one without acknowledging the other is a failure."
  }]
}

▶ ② Execution Instructions 144 chars

Use ONLY MCP Knowledge Brain retrieval tools with domain="brain-test".
Question: {{QUESTION}}
Return ONLY this JSON:
{{REQUIRED_ANSWER_SCHEMA}}

⚖ ③ Judge Instructions 338 chars

Ground truth:
{{SETUP_CONTEXT_JSON}}

Tested AI answer:
{{EXEC_ANSWER_JSON}}

Two people share the same title. The question is ambiguous. Must cite both and use UNCERTAIN.
Critical failures: named only one person without noting the ambiguity, cited only one source, hallucinated source_id.

Return ONLY this JSON:
{{JUDGE_RESULT_SCHEMA}}

Critical Failure Conditions

named_only_one_without_noting_ambiguity
cited_only_one_source
hallucinated_source_id
wrong_label_not_uncertain

Recent Run History

3 runs

When	Run ID	Pass Rate	Avg Score	Reps
2026-05-24 13:08	20260524T130808Z-kqze	0%	50.0	0/1	View →
2026-05-24 12:41	20260524T124148Z-z2do	100%	86.0	1/1	View →
2026-05-24 11:37	20260524T113756Z-kduj	100%	95.0	1/1	View →

📄 Raw YAML cases/entity_disambiguation/ENT-003.yaml

schema_version: "1.0"
test_id: "ENT-003"
category: "entity_disambiguation"
severity: "critical"
repetitions: 3
reasoning_type: "UNCERTAIN"
num_documents: 2
num_questions: 1
tags: ["entity-disambiguation", "ambiguous-reference", "pronoun-plus-entity"]

setup_instructions: |
  Generate two documents about two people with the same job title but different names.
  Ask a question using the job title (not the name) — the answer is ambiguous.

  Requirements:
  - Doc A: "<Name-A> is the <Title> of <Organization-A>."
  - Doc B: "<Name-B> is the <Title> of <Organization-B>."
  - Same job title, different names and organizations.
  - source_ids: KB-{{RUN}}-ENT-003-A-v1 and KB-{{RUN}}-ENT-003-B-v1.
  - Question: "Who is the <Title>?" — ambiguous because both are <Title> of different orgs.

  Output ONLY this JSON:
  {
    "run_id": "{{RUN}}",
    "test_id": "ENT-003",
    "repetition": {{REP}},
    "documents": [
      {"source_id": "KB-{{RUN}}-ENT-003-A-v1", "content": "<Name-A> is the <Title> of <Organization-A>.", "title": "<title>", "version": 1},
      {"source_id": "KB-{{RUN}}-ENT-003-B-v1", "content": "<Name-B> is the <Title> of <Organization-B>.", "title": "<title>", "version": 1}
    ],
    "questions": ["Who is the <Title>?"],
    "expected_answers": [{
      "question_index": 0,
      "correct_answer": null,
      "required_source_ids": ["KB-{{RUN}}-ENT-003-A-v1", "KB-{{RUN}}-ENT-003-B-v1"],
      "required_quote": null,
      "required_label": "UNCERTAIN",
      "must_refuse": false,
      "notes": "Two people hold the same title in different orgs. The question is ambiguous — must surface both and note the ambiguity. Picking one without acknowledging the other is a failure."
    }]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools with domain="brain-test".
  Question: {{QUESTION}}
  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}

judge_instructions: |
  Ground truth:
  {{SETUP_CONTEXT_JSON}}

  Tested AI answer:
  {{EXEC_ANSWER_JSON}}

  Two people share the same title. The question is ambiguous. Must cite both and use UNCERTAIN.
  Critical failures: named only one person without noting the ambiguity, cited only one source, hallucinated source_id.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}

critical_failures:
  - "named_only_one_without_noting_ambiguity"
  - "cited_only_one_source"
  - "hallucinated_source_id"
  - "wrong_label_not_uncertain"