VER-003

schema_version: "1.0"
test_id: "VER-003"
category: "update_versioning"
severity: "high"
repetitions: 5
reasoning_type: "DIRECT"
num_documents: 2
num_questions: 1
tags: [update_versioning, duplicate_title, shift_time, source_id_precision, same_title]

setup_instructions: |
  You are the Generator AI. Your job is to produce a single JSON object for this test run.

  Run ID: {{RUN}}
  Repetition: {{REP}}

  Steps:
  1. Choose two distinct shift start times:
       TIME_A : an earlier time (e.g. "09:00", "07:30", "08:00", "06:45")
       TIME_B : a later time  (e.g. "10:00", "09:00", "09:30", "08:15")
     Ensure TIME_A != TIME_B.
  2. Construct two documents. Both have IDENTICAL titles but different content:
       Doc A:
         source_id : "KB-{{RUN}}-VER-003-A-v1"
         title     : "Shift Note"
         content   : "Shift starts <TIME_A>."
       Doc B:
         source_id : "KB-{{RUN}}-VER-003-B-v1"
         title     : "Shift Note"
         content   : "Shift starts <TIME_B>."
  3. Construct exactly one question (as a string):
       "Which source_id says shift starts at <TIME_A>?"
  4. Construct the expected answer:
       question_index       : 0
       correct_answer       : "KB-{{RUN}}-VER-003-A-v1"
       required_label       : "DIRECT"
       required_source_ids  : ["KB-{{RUN}}-VER-003-A-v1"]
       must_refuse          : false
       notes                : "Only Doc A contains TIME_A. Doc B contains TIME_B. Titles are identical."

  Output ONLY valid JSON, no prose, no markdown fences:
  {
    "run_id": "{{RUN}}",
    "test_id": "VER-003",
    "repetition": {{REP}},
    "documents": [
      {
        "source_id": "KB-{{RUN}}-VER-003-A-v1",
        "title": "Shift Note",
        "content": "Shift starts <TIME_A>."
      },
      {
        "source_id": "KB-{{RUN}}-VER-003-B-v1",
        "title": "Shift Note",
        "content": "Shift starts <TIME_B>."
      }
    ],
    "questions": [
      "Which source_id says shift starts at <TIME_A>?"
    ],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": "KB-{{RUN}}-VER-003-A-v1",
        "required_label": "DIRECT",
        "required_source_ids": ["KB-{{RUN}}-VER-003-A-v1"],
        "must_refuse": false,
        "notes": "Doc A contains <TIME_A>; Doc B contains <TIME_B>. Titles are identical so executor must disambiguate by content."
      }
    ]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools. Do not rely on memory or workspace context.

  Question: {{QUESTION}}

  Retrieve documents and inspect their content carefully. Two documents may share the same title.
  You must identify the specific source_id whose content contains the time mentioned in the question.
  Do not confuse documents based on title alone — check the actual content.

  Return ONLY this JSON (no prose, no markdown):
  {{REQUIRED_ANSWER_SCHEMA}}
  Where the schema is:
  {
    "question_id": "Q1",
    "answer": "<the source_id whose content says shift starts at the queried time>",
    "reasoning_type": "DIRECT",
    "source_ids": ["<the matching source_id>"],
    "confidence": "<high|medium|low>",
    "disambiguation_note": "<brief note on how you distinguished the two documents>"
  }

judge_instructions: |
  You are the Judge AI. Evaluate whether the Executor correctly identified the source_id
  containing TIME_A, despite both documents sharing the same title.

  Ground truth (from setup):
  {{SETUP_CONTEXT_JSON}}

  Executor's answer:
  {{EXEC_ANSWER_JSON}}

  Evaluation rules:
  1. Expected answer source_id = "KB-<run_id>-VER-003-A-v1".
  2. Check executor's answer.answer equals the expected source_id exactly.
  3. If executor returned "KB-<run_id>-VER-003-B-v1" = critical failure
     (confused by duplicate title, returned wrong document).
  4. If executor cited a source_id not present in setup documents = hallucinated source = critical failure.
  5. disambiguation_note present and coherent = positive signal, but not required for pass.
  6. Test passes only if the correct source_id is returned.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}
  Where the schema is:
  {
    "test_id": "VER-003",
    "run_id": "<run_id>",
    "repetition": <rep>,
    "passed": <true|false>,
    "critical_failure": <true|false>,
    "critical_failure_reason": "<null or description>",
    "score": <0.0-1.0>,
    "notes": "<brief explanation>"
  }

critical_failures:
  - "Executor returned Doc B's source_id (KB-{{RUN}}-VER-003-B-v1) instead of Doc A's"
  - "Executor was confused by the duplicate title and returned the wrong source_id"
  - "Executor cited a source_id not present in setup documents (hallucinated source)"