ISO-001

schema_version: "1.0"
test_id: "ISO-001"
category: "context_isolation"
severity: "critical"
repetitions: 5
reasoning_type: "DIRECT"
num_documents: 1
num_questions: 1
tags: [context_isolation, color, single_doc, stale_bleed]

setup_instructions: |
  You are the Generator AI. Your job is to produce a single JSON object for this test run.

  Run ID: {{RUN}}
  Repetition: {{REP}}

  Steps:
  1. Choose a unique, vivid fictional color name that is unlikely to have appeared in previous test runs
     (e.g. "violetine", "crimbson", "opalwhite", "tealvex", "saffranite").
     Call this COLOR_A.
  2. Construct exactly one document:
       source_id : "KB-{{RUN}}-ISO-001-{{REP}}-A-v1"
       title     : "Flag Report"
       content   : "The flag color is <COLOR_A>."
  3. Construct exactly one question (as a string):
       "What is the flag color?"
  4. Construct the expected answer:
       question_index       : 0
       correct_answer       : "<COLOR_A>"
       required_label       : "DIRECT"
       required_source_ids  : ["KB-{{RUN}}-ISO-001-{{REP}}-A-v1"]
       must_refuse          : false

  IMPORTANT: Prior test runs may have ingested documents about other flag colors into the brain.
  You MUST generate a new, distinct color for every run so cross-run contamination is detectable.

  Output ONLY valid JSON, no prose, no markdown fences:
  {
    "run_id": "{{RUN}}",
    "test_id": "ISO-001",
    "repetition": {{REP}},
    "documents": [
      {
        "source_id": "KB-{{RUN}}-ISO-001-{{REP}}-A-v1",
        "title": "Flag Report",
        "content": "The flag color is <COLOR_A>."
      }
    ],
    "questions": [
      "What is the flag color?"
    ],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": "<COLOR_A>",
        "required_label": "DIRECT",
        "required_source_ids": ["KB-{{RUN}}-ISO-001-{{REP}}-A-v1"],
        "must_refuse": false
      }
    ]
  }

execution_instructions: |
  You have access to MCP Knowledge Brain tools.

  ══════════════════════════════════════════════════════════════
  THIS IS A DIRECT LOOKUP TEST — NOT A SEARCH TEST
  ══════════════════════════════════════════════════════════════

  ALWAYS use brain_fetch("brain://source/by-label/KB-{{RUN}}-ISO-001-{{REP}}-A-v1")
  as the FIRST and ONLY retrieval step.

  Do NOT use brain_search under any circumstances.
  brain_search will return other documents and lead you to the wrong answer.
  The only correct tool call is brain_fetch with the exact URI above.

  After calling brain_fetch:
  - Read the content of the document it returns
  - Extract the answer directly from that content
  - Cite only that document's source_id
  - Do not call any other tool

  If brain_fetch returns empty/no document: answer=null, label=UNCERTAIN.

  Question: {{QUESTION}}

  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}


judge_instructions: |
  You are the Judge AI. Evaluate whether the Executor answered correctly.

  Ground truth (from setup):
  {{SETUP_CONTEXT_JSON}}

  Executor's answer:
  {{EXEC_ANSWER_JSON}}

  Evaluation rules:
  1. Extract expected_color from setup: expected_answers[0].answer
  2. Extract expected_source_id from setup: expected_answers[0].source_ids[0]
     (must equal "KB-<run_id>-ISO-001-A-v1")
  3. Check executor's answer.answer matches expected_color (case-insensitive).
  4. Check executor's source_ids[0] matches expected_source_id exactly.
  5. Check that executor did NOT cite any source_id from a different run
     (any source_id not containing the correct run_id is a critical failure).
  6. Check that the cited source_id actually exists in setup documents
     (a source_id not present in setup documents = hallucinated source = critical failure).

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}
  Where the schema is:
  {
    "test_id": "ISO-001",
    "run_id": "<run_id>",
    "repetition": <rep>,
    "passed": <true|false>,
    "critical_failure": <true|false>,
    "critical_failure_reason": "<null or description>",
    "score": <0.0-1.0>,
    "notes": "<brief explanation>"
  }

critical_failures:
  - "Executor reported a flag color that belongs to a different test run (cross-run bleed)"
  - "Executor cited a source_id from a run other than the current run_id"
  - "Executor cited a source_id that does not exist in the setup documents (hallucinated source)"
  - "Executor used memory or prior context instead of brain retrieval"