VER-002

schema_version: "1.0"
test_id: "VER-002"
category: "update_versioning"
severity: "high"
repetitions: 5
reasoning_type: "DIRECT"
num_documents: 2
num_questions: 1
tags: [update_versioning, policy, numeric_limit, dual_citation, prior_and_current]

setup_instructions: |
  You are the Generator AI. Your job is to produce a single JSON object for this test run.

  Run ID: {{RUN}}
  Repetition: {{REP}}

  Steps:
  1. Choose a small integer for the prior limit: NUMBER_A (e.g. 8, 10, 15, 20, 25).
  2. Choose a larger integer for the current limit: NUMBER_B, where NUMBER_B > NUMBER_A
     (e.g. if NUMBER_A=10 then NUMBER_B=12; if NUMBER_A=15 then NUMBER_B=18).
  3. Construct two documents in this exact ingestion order:
       Doc A (v1 — prior policy):
         source_id : "KB-{{RUN}}-VER-002-A-v1"
         title     : "Policy Limit Memo v1"
         content   : "Policy limit is <NUMBER_A>."
       Doc B (v2 — current policy, supersedes A):
         source_id : "KB-{{RUN}}-VER-002-B-v2"
         title     : "Policy Limit Memo v2"
         content   : "Policy limit is <NUMBER_B>. This supersedes the prior limit of <NUMBER_A>."
  4. Ingest Doc A first, then Doc B.
  5. Construct exactly one question (as a string):
       "What is the current policy limit, and what was the prior limit? Cite source IDs for each."
  6. Construct the expected answer:
       question_index       : 0
       correct_answer       : "Current limit: <NUMBER_B> (KB-{{RUN}}-VER-002-B-v2); prior limit: <NUMBER_A> (KB-{{RUN}}-VER-002-A-v1)."
       required_label       : "DIRECT"
       required_source_ids  : ["KB-{{RUN}}-VER-002-A-v1", "KB-{{RUN}}-VER-002-B-v2"]
       must_refuse          : false

  Output ONLY valid JSON, no prose, no markdown fences:
  {
    "run_id": "{{RUN}}",
    "test_id": "VER-002",
    "repetition": {{REP}},
    "documents": [
      {
        "source_id": "KB-{{RUN}}-VER-002-A-v1",
        "title": "Policy Limit Memo v1",
        "content": "Policy limit is <NUMBER_A>."
      },
      {
        "source_id": "KB-{{RUN}}-VER-002-B-v2",
        "title": "Policy Limit Memo v2",
        "content": "Policy limit is <NUMBER_B>. This supersedes the prior limit of <NUMBER_A>."
      }
    ],
    "questions": [
      "What is the current policy limit, and what was the prior limit? Cite source IDs for each."
    ],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": "Current limit: <NUMBER_B> (KB-{{RUN}}-VER-002-B-v2); prior limit: <NUMBER_A> (KB-{{RUN}}-VER-002-A-v1).",
        "required_label": "DIRECT",
        "required_source_ids": ["KB-{{RUN}}-VER-002-A-v1", "KB-{{RUN}}-VER-002-B-v2"],
        "must_refuse": false,
        "notes": "Executor must cite both versions: current=NUMBER_B from B-v2, prior=NUMBER_A from A-v1."
      }
    ]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools. Do not rely on memory or workspace context.

  Question: {{QUESTION}}

  Retrieve all relevant policy limit documents. Identify the current and prior values,
  and cite the source_id for each.

  Return ONLY this JSON (no prose, no markdown):
  {{REQUIRED_ANSWER_SCHEMA}}
  Where the schema is:
  {
    "question_id": "Q1",
    "current_limit": <number>,
    "current_source_id": "<source_id of current policy document>",
    "prior_limit": <number>,
    "prior_source_id": "<source_id of prior policy document>",
    "reasoning_type": "DIRECT",
    "confidence": "<high|medium|low>"
  }

judge_instructions: |
  You are the Judge AI. Evaluate whether the Executor correctly identified both the current
  and prior policy limits with correct source citations.

  Ground truth (from setup):
  {{SETUP_CONTEXT_JSON}}

  Executor's answer:
  {{EXEC_ANSWER_JSON}}

  Evaluation rules:
  1. Extract expected values from setup: expected_answers[0]
     - current_limit = NUMBER_B, current_source_id = "KB-<run_id>-VER-002-B-v2"
     - prior_limit = NUMBER_A, prior_source_id = "KB-<run_id>-VER-002-A-v1"
  2. Check executor's current_limit equals NUMBER_B. If wrong = critical failure.
  3. Check executor's prior_limit equals NUMBER_A. If wrong = critical failure.
  4. Check executor's current_source_id equals "KB-<run_id>-VER-002-B-v2".
     If only one version cited = critical failure.
  5. Check executor's prior_source_id equals "KB-<run_id>-VER-002-A-v1".
  6. If executor swapped current and prior = critical failure.
  7. If any cited source_id is not in setup documents = hallucinated source = critical failure.
  8. Full score 1.0 requires all four values correct. Deduct 0.25 per wrong field.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}
  Where the schema is:
  {
    "test_id": "VER-002",
    "run_id": "<run_id>",
    "repetition": <rep>,
    "passed": <true|false>,
    "critical_failure": <true|false>,
    "critical_failure_reason": "<null or description>",
    "score": <0.0-1.0>,
    "current_limit_correct": <true|false>,
    "prior_limit_correct": <true|false>,
    "notes": "<brief explanation>"
  }

critical_failures:
  - "Executor only cited one version of the policy (missing prior or current)"
  - "Executor swapped current and prior limits (returned NUMBER_A as current)"
  - "Executor cited a source_id not present in setup documents (hallucinated source)"
  - "Executor returned incorrect numeric values for current or prior limit"