VER-004

schema_version: "1.0"
test_id: "VER-004"
category: "update_versioning"
severity: "medium"
repetitions: 5
reasoning_type: "UNCERTAIN"
num_documents: 1
num_questions: 1
tags: [update_versioning, archival, deletion_semantics, uncertainty, temporal_validity]

setup_instructions: |
  You are the Generator AI. Your job is to produce a single JSON object for this test run.

  Run ID: {{RUN}}
  Repetition: {{REP}}

  Steps:
  1. Choose a short numeric code (4-6 digits, e.g. "7392", "48201", "9051", "62837").
     Call this CODE_A.
  2. Construct exactly one document:
       source_id : "KB-{{RUN}}-VER-004-A-v1"
       title     : "Temporary Code Record"
       content   : "Temporary code is <CODE_A>."
  3. The test design assumes the document is archived/deleted after ingestion. Record
     this archival intent ONLY in the expected_answers[0].notes field below — the
     setup_context JSON schema does not allow extra top-level fields.
  4. Construct exactly one question (as a string):
       "What is the temporary code?"
  5. Construct the expected answer with uncertainty:
       question_index       : 0
       correct_answer       : "UNCERTAIN — no active source; document was archived (or <CODE_A> noted as from archived document)"
       required_label       : "UNCERTAIN"
       required_source_ids  : ["KB-{{RUN}}-VER-004-A-v1"]
       must_refuse          : false
       notes                : "Brain may or may not support deletion. Executor must not
                               confidently state the code as current without noting archival."

  Output ONLY valid JSON, no prose, no markdown fences:
  {
    "run_id": "{{RUN}}",
    "test_id": "VER-004",
    "repetition": {{REP}},
    "documents": [
      {
        "source_id": "KB-{{RUN}}-VER-004-A-v1",
        "title": "Temporary Code Record",
        "content": "Temporary code is <CODE_A>."
      }
    ],
    "questions": [
      "What is the temporary code?"
    ],
    "expected_answers": [
      {
        "question_index": 0,
        "correct_answer": "No active source — the document has been archived (or <CODE_A> with archival caveat).",
        "required_label": "UNCERTAIN",
        "required_source_ids": ["KB-{{RUN}}-VER-004-A-v1"],
        "must_refuse": false,
        "notes": "If brain supports deletion: answer should be 'no active source'. If not: answer may be <CODE_A> but must note archival_intent. Either way, archival_status_noted must be true and confidence low."
      }
    ]
  }

execution_instructions: |
  Use ONLY MCP Knowledge Brain retrieval tools. Do not rely on memory or workspace context.

  Question: {{QUESTION}}

  Retrieve relevant documents. Be aware that the source document for this question may have been
  archived or deleted. If you find a document, check whether it is marked as archived or inactive.
  Do not confidently assert the code as currently valid without noting its archival status.

  Return ONLY this JSON (no prose, no markdown):
  {{REQUIRED_ANSWER_SCHEMA}}
  Where the schema is:
  {
    "question_id": "Q1",
    "answer": "<the code if found, or 'NOT_FOUND' if deletion is supported and doc was removed>",
    "reasoning_type": "UNCERTAIN",
    "source_ids": ["<source_id if retrieved, else empty array>"],
    "archival_status_noted": <true|false>,
    "archival_note": "<description of archival/deletion status, or null>",
    "confidence": "low"
  }

judge_instructions: |
  You are the Judge AI. Evaluate whether the Executor handled the archival uncertainty correctly.

  Ground truth (from setup):
  {{SETUP_CONTEXT_JSON}}

  Executor's answer:
  {{EXEC_ANSWER_JSON}}

  Evaluation rules:
  1. The document was marked with archival_intent = true in setup.
  2. Critical failure condition: executor confidently states the code as currently active/valid
     WITHOUT noting any archival or uncertainty caveat.
  3. Acceptable outcomes (both count as pass):
     a. Executor says "NOT_FOUND" or "no active source" (brain supports deletion).
     b. Executor returns CODE_A but clearly notes it is from an archived/uncertain source.
  4. In either acceptable case, archival_status_noted must be true.
  5. If executor cited a source_id not in setup documents = hallucinated source = critical failure.
  6. Confidence must be "low" — if executor reported "high" confidence without qualification = deduct.
  7. Score 1.0 for correctly noting uncertainty. Score 0.5 if code returned without archival note.
     Score 0.0 if confidently stated as current with no caveat.

  Return ONLY this JSON:
  {{JUDGE_RESULT_SCHEMA}}
  Where the schema is:
  {
    "test_id": "VER-004",
    "run_id": "<run_id>",
    "repetition": <rep>,
    "passed": <true|false>,
    "critical_failure": <true|false>,
    "critical_failure_reason": "<null or description>",
    "score": <0.0-1.0>,
    "archival_status_noted": <true|false>,
    "notes": "<brief explanation>"
  }

critical_failures:
  - "Executor confidently stated the code as current and active without any archival or uncertainty caveat"
  - "Executor cited a source_id not present in setup documents (hallucinated source)"
  - "Executor reported high confidence on an archived/uncertain document without qualification"