Plan: the model battery — every shipped model, per role, with a matrix (#79)

Which of the nineteen models @gadhs/pi ships is fit for which role — the reviewer, the judge, a read-only helper, the main coder, the plan reviewer — measured the same way every time, with the numbers published where a team can read them before it deviates from the defaults.

Two of these measurements exist and were done once by hand (tools/reviewer-eval, #73/#74) or once per rubric change (packages/pi-modes/eval). They become two roles of one runner, and the hand-scoring becomes a scorer.

tools/model-battery/:

tools/reviewer-eval/ ceases to exist except results/ (the 2026-09-07 record the calibration reads); nothing is symlinked.

Per call, recorded from what the wire returns: wall ms, usage (input/output/cacheRead/reasoning), usage.cost (pi-ai computes it from the catalog rate, so the cost column is the catalog’s), stop reason, error text. Per model, from the catalog: window, max output, rates, publisher kind. meta.json records the run’s pi version, pi-ai version, the repo commit, the Explore worktree commit, and every model ref with its thinking level.

Erratum (2026-09-08): the headless path first spawned synchronously, which ran the lanes one at a time; it is asynchronous, and a relaunch into the same --out directory keeps records already on disk (resume), which the design did not name. A run in phases into one directory merges meta.json. Explore counts yield rejections apart from tool failures (the contract’s retry ladder), and every role reports capped (length stops) so a budget-starved cell reads as such (#81).

The tool-less roles call the extension’s own streamSimple in-process: factory({ registerProvider }) yields the three providers exactly as pi gets them, so the request is production’s and there is no ten-second pi cold start per call (2,300 judge calls across the catalog would otherwise be a day). The tool-using roles need pi’s agent loop and go through pi -p --mode json --no-extensions -e <vertex extension> --tools …; the JSON stream gives usage, tool executions and their errors.

The reviewer and judge prompts are IMPORTED from the shipped code, as reviewer-eval does today; the Explore body is read from the packaged agent file at run time. The battery never carries its own copy of a production prompt - pinned by test/battery.test.ts: the runner’s source imports buildReviewPrompt/buildPlanReviewPrompt from pi-workflow and runJudge from pi-modes, and contains no line of either system prompt.

A case whose scoring needs a human is not in the battery. The reviewer scorer is the new piece: each planted defect in key.json gains anchors: { files: […​], lines: [from, to], terms: […​] } (written once, by hand, from the case trees). Before any new run, score.mjs is run over the outputs already on disk from 2026-09-07 (results/2026-09-07T15-55-01, four models, 36 defects) and its verdicts diffed against `scores.json’s hand scores. Disagreements are anchor bugs until proven otherwise; the calibration diff is committed beside the anchors. The bar: at most two cells of 144 disagree, each explained in the commit.

The explore and coder scorers are exact by construction (paths present, tests passed). The judge scorer is the existing harness’s.

--runs N (default 2) repeats every case; the summary carries min, max and mean per cell, and the page prints the spread beside the mean. pi exposes no temperature control per call and the battery adds none: "deterministic" means fixed inputs, fixed prompts, a pinned commit, a recorded environment - the model’s variance is measured, not hidden.

matrix.json holds one rule per role, applied to summary.json by render.mjs:

Every cell prints fit / usable / no and the numbers that decided it, and within a fit column models are ordered by cost per case. The shipped defaults (auto Opus 5, plan Fable, judge Haiku, reviewer Fable, Explore/Verify Haiku, Research Sonnet) are starred so the page argues them from the same table. Changing a default is never this plan’s business: each is its own issue, citing the page.

render.mjs writes docs/modules/ROOT/pages/model-battery.adoc from the latest summary.json: a banner naming the run, date, commit, pi version and cost; a fixed method preamble (from tools/model-battery/page-head.adoc, hand-written); one table per role; the matrix; a "what the defaults are and why" section computed from the starred rows. The page header says it is generated and where to edit. reviewer-eval.adoc keeps its history and gains a pointer; tools/reviewer-eval/run.mjs and make-cases.py are absorbed (the cases move under the battery; the old runner is deleted, not aliased).

A full run over nineteen models at --runs 2: the coder and explore roles on Opus 5 and Fable dominate (agent loops of 100-300k tokens per task); the estimate is US$150-250, most of it those two models. The runner prints the running cost and a per-model subtotal; --model and --role are the normal way to run it, and the page names the subset a run covered. The first full run is this plan’s step 8 and its cost is recorded in the closing comment.