ISO-004

schema_version: "1.0"
test_id: "ISO-004"
category: "context_isolation"
severity: "high"
repetitions: 5
reasoning_type: "DIRECT"
num_documents: 1
num_questions: 1
tags: [context_isolation, ownership, single_doc, no_cross_case, source_discipline]

setup_instructions: |
  You are the Generator AI. Your job is to produce a single JSON object for this test run.

  Run ID: {{RUN}}
  Repetition: {{REP}}

  Steps:
  1. Choose a fictional person's first name (e.g. "Marvella", "Tobsin", "Quelda", "Ruffwick", "Zinora").
     Call this PERSON_A.
  2. Choose a vivid color not used in other ISO tests this run (e.g. "amber", "indigo", "chartreuse").
     Call this COLOR_A.
  3. Choose a small, concrete object (e.g. "kite", "lantern", "compass", "satchel", "prism").
     Call this OBJECT_A.
  4. Construct exactly one document:
       source_id : "KB-{{RUN}}-ISO-004-A-v1"
       title     : "Ownership Record"
       content   : "<PERSON_A> owns the <COLOR_A> <OBJECT_A>."
  5. Construct exactly one question (as a string):
       "Who owns the <COLOR_A> <OBJECT_A>?"
  6. Construct the expected answer:
       question_index       : 0
       correct_answer       : "<PERSON_A>"
       required_label       : "DIRECT"
       required_source_ids  : ["KB-{{RUN}}-ISO-004-A-v1"]
       must_refuse          : false

  Output ONLY valid JSON, no prose, no markdown fences:
  {
    "run_id": "{{RUN}}",
    "test_id": "ISO-004",
    "repetition": {{REP}},
    "documents": [
      {
        "source_id": "KB-{{RUN}}-ISO-004-A-v1",
        "title": "Ownership Record",
        "content": "<PERSON_A> owns the <COLOR_A> <OBJECT_A>."
      }
    ],
    "questions": [
      "Who owns the <COLOR_A> <OBJECT_A>?"
    ],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": "<PERSON_A>",
        "required_label": "DIRECT",
        "required_source_ids": ["KB-{{RUN}}-ISO-004-A-v1"],
        "must_refuse": false
      }
    ]
  }

execution_instructions: |
  You have access to MCP Knowledge Brain tools.

  MANDATORY: Use brain_fetch to retrieve the document DIRECTLY by its source label.
  Do NOT use brain_search — it will return noise documents and distract you.

  Step 1: Call brain_fetch with URI: brain://source/by-label/KB-{{RUN}}-ISO-004-A-v1
  Step 2: Read the content of the returned document
  Step 3: Extract the value asked for in the question (e.g. the flag color, animal, etc.)
  Step 4: Return that value as your answer, citing the source_id from the fetched document

  If brain_fetch returns no document for that label: answer null, label UNCERTAIN.
  Do NOT fall back to brain_search. Do NOT cite any source other than the fetched document.

  Question: {{QUESTION}}

  Return ONLY this JSON:
  {{REQUIRED_ANSWER_SCHEMA}}


judge_instructions: |
  You are the Judge AI. Evaluate whether the Executor answered correctly.

  Ground truth (from setup):
  {{SETUP_CONTEXT_JSON}}

  Executor's answer:
  {{EXEC_ANSWER_JSON}}

  Evaluation rules:
  1. Extract expected_person from setup: expected_answers[0].answer
  2. Extract expected_source_id from setup: expected_answers[0].source_ids[0]
     (must equal "KB-<run_id>-ISO-004-A-v1")
  3. Check executor's answer.answer matches expected_person (case-insensitive).
  4. Check executor's source_ids contains only the expected_source_id.
  5. If executor cited any source_id that does NOT start with "KB-<run_id>-ISO-004"
     = referenced docs outside current test = critical failure.
  6. If executor cited a source_id not present in setup documents = hallucinated source = critical failure.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}
  Where the schema is:
  {
    "test_id": "ISO-004",
    "run_id": "<run_id>",
    "repetition": <rep>,
    "passed": <true|false>,
    "critical_failure": <true|false>,
    "critical_failure_reason": "<null or description>",
    "score": <0.0-1.0>,
    "notes": "<brief explanation>"
  }

critical_failures:
  - "Executor cited a source_id from outside the current test case (cross-case contamination)"
  - "Executor cited a source_id not present in setup documents (hallucinated source)"
  - "Executor returned a person name not matching the current run's document"