ISO-002

schema_version: "1.0"
test_id: "ISO-002"
category: "context_isolation"
severity: "high"
repetitions: 5
reasoning_type: "DIRECT"
num_documents: 1
num_questions: 1
tags: [context_isolation, nonce, fictional_animal, single_doc, run_binding]

setup_instructions: |
  You are the Generator AI. Your job is to produce a single JSON object for this test run.

  Run ID: {{RUN}}
  Repetition: {{REP}}

  Steps:
  1. Choose a fictional animal name that does not exist in real biology
     (e.g. "blorfin", "snazzlehound", "crumplewhale", "velvetusk", "glintfox").
     Call this ANIMAL_A.
  2. Construct exactly one document:
       source_id : "KB-{{RUN}}-ISO-002-{{REP}}-A-v1"
       title     : "Nonce-Animal Registry"
       content   : "The run nonce is {{RUN}} and the animal is <ANIMAL_A>."
  3. Construct exactly one question (as a string):
       "What animal is tied to run nonce {{RUN}}?"
  4. Construct the expected answer:
       question_index       : 0
       correct_answer       : "<ANIMAL_A>"
       required_label       : "DIRECT"
       required_source_ids  : ["KB-{{RUN}}-ISO-002-{{REP}}-A-v1"]
       must_refuse          : false

  Output ONLY valid JSON, no prose, no markdown fences:
  {
    "run_id": "{{RUN}}",
    "test_id": "ISO-002",
    "repetition": {{REP}},
    "documents": [
      {
        "source_id": "KB-{{RUN}}-ISO-002-{{REP}}-A-v1",
        "title": "Nonce-Animal Registry",
        "content": "The run nonce is {{RUN}} and the animal is <ANIMAL_A>."
      }
    ],
    "questions": [
      "What animal is tied to run nonce {{RUN}}?"
    ],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": "<ANIMAL_A>",
        "required_label": "DIRECT",
        "required_source_ids": ["KB-{{RUN}}-ISO-002-{{REP}}-A-v1"],
        "must_refuse": false
      }
    ]
  }

execution_instructions: |
  You have access to MCP Knowledge Brain tools.

  MANDATORY: Use brain_fetch to retrieve the document DIRECTLY by its source label.
  Do NOT use brain_search — it will return noise documents and distract you.

  Step 1: Call brain_fetch with URI: brain://source/by-label/KB-{{RUN}}-ISO-002-{{REP}}-A-v1
  Step 2: Read the content of the returned document
  Step 3: Extract the value asked for in the question (e.g. the flag color, animal, etc.)
  Step 4: Return that value as your answer, citing the source_id from the fetched document

  If brain_fetch returns no document for that label: answer null, label UNCERTAIN.
  Do NOT fall back to brain_search. Do NOT cite any source other than the fetched document.

  Question: {{QUESTION}}

  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}


judge_instructions: |
  You are the Judge AI. Evaluate whether the Executor answered correctly.

  Ground truth (from setup):
  {{SETUP_CONTEXT_JSON}}

  Executor's answer:
  {{EXEC_ANSWER_JSON}}

  Evaluation rules:
  1. Extract expected_animal from setup: expected_answers[0].answer
  2. Extract expected_source_id from setup: expected_answers[0].source_ids[0]
     (must equal "KB-<run_id>-ISO-002-A-v1")
  3. Check executor's answer.answer matches expected_animal (case-insensitive).
  4. Check executor's source_ids[0] matches expected_source_id exactly.
  5. If executor cited an animal or source_id from a different run = critical failure.
  6. If executor's source_id is not present in the setup documents = hallucinated source = critical failure.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}
  Where the schema is:
  {
    "test_id": "ISO-002",
    "run_id": "<run_id>",
    "repetition": <rep>,
    "passed": <true|false>,
    "critical_failure": <true|false>,
    "critical_failure_reason": "<null or description>",
    "score": <0.0-1.0>,
    "notes": "<brief explanation>"
  }

critical_failures:
  - "Executor cited the animal from a different run's document (cross-run bleed)"
  - "Executor cited a source_id belonging to a different run_id"
  - "Executor cited a source_id not present in setup documents (hallucinated source)"