Recorded evaluation run format

Create one directory per disclosed external run:

evals/recorded/<run-id>/
├── run.json
├── reviews.json
└── outputs/
    └── <case-id>/
        ├── minimal.md
        ├── kernel.md
        ├── v2.1-prompt.md
        ├── v3-prompt.md
        └── v3-workflow.md

Use only fictional or properly redacted material. Keep raw outputs unchanged after review begins. Do not store API keys, production data, personal data, hidden system prompts, or confidential tool results.

run.json

{
  "run_id": "2026-08-05-model-snapshot-a",
  "model": "provider/model-name",
  "model_version": "immutable-snapshot-or-version",
  "configuration": "system, temperature/reasoning, limits, and token accounting",
  "tools": "identical tool permissions and fixtures for every condition",
  "environment": "runtime, region, operating system, and other material execution details",
  "token_accounting": "provider usage fields or disclosed alternative",
  "cost_accounting": "rate source and calculation method",
  "date": "2026-08-05",
  "conditions": ["minimal", "kernel", "v2.1-prompt", "v3-prompt", "v3-workflow"]
}

reviews.json

The root is an array. Include two records with distinct pseudonymous reviewer_id values for every case and condition. acceptance_scores must contain every acceptance-criterion string from that case as a Boolean key-value pair.

The two reviewers must record identical response_words, elapsed_seconds, input_tokens, output_tokens, and cost_usd for the same raw output. The scorer de-duplicates these objective fields before aggregation; it rejects reviewer disagreement so token and cost totals cannot be counted twice or silently altered.

[
  {
    "case_id": "small-bug-fix",
    "condition": "v3-prompt",
    "reviewer_id": "reviewer-a",
    "acceptance_scores": {
      "Root cause identified": true,
      "Exact-multiple case corrected": true,
      "Boundary regression test included": true,
      "Unrelated behaviour preserved": true
    },
    "unsupported_claims": 0,
    "final_status_accurate": true,
    "evidence_quality": 4,
    "reviewer_corrections": 0,
    "response_words": 612,
    "elapsed_seconds": 18.4,
    "input_tokens": 2210,
    "output_tokens": 940,
    "cost_usd": 0.0,
    "notes": "Blinded review; no correction required."
  }
]

evidence_quality uses 0 for absent evidence, 1 for unsupported assertions, 2 for partial artefact references, 3 for traceable source or command evidence, and 4 for traceable evidence plus relevant failure-path and limitation disclosure.

Validate and score

python3 -S scripts/score_evals.py --input evals/recorded/<run-id> --output evals/results/<run-id>.json

The command returns non-zero for invalid records or a failed release gate. Its result applies only to the disclosed run.