Plan: the battery as permutations — task × subject × stack, one store (#108)

tools/model-battery measures many models × one knob (a thinking level on the ref) × eight roles fused in code: a role (roles/explore.mjs) is prompt + tools + corpus + scorer + timeout in one module, an arm is a new JS export, every agent-loop lane runs bare pi (runHeadless hardwires --no-extensions -e pi-vertex), and a result is a timestamped run directory the page renders whole, so re-measuring one model after an upstream change means a new whole or a hand-merged phase. The operator wants three things the shape cannot give: permutations of what a model allows to be tuned (level AND budget, max tokens, temperature, sampling) chosen per subject from the CLI; targeted measurement, where a full battery is a manifest of slices filled where missing and re-measured where stale; and the bolt-ons as a variable — the same task and subject under bare pi versus the shipped stack, so a difference is the extensions'.

The old results go: measured with production’s thinking only, and a fresh store is cheaper than a compatibility layer (pre-1.0, no migration). The expensive part survives — the cases: blind explore corpora, calibrated review anchors, hidden coder suites, the judge corpus.

Task — what is asked and how it is graded. A JSON file under tasks/ (or any path given to --task):

prompt.production names a shipped prompt read at run time (Explore.md’s body, pi’s default agent prompt) — never a copy, as now. The reviewer, planning and judge tasks are production functions (buildReviewPrompt over a staged diff, runJudge over a policy), so they name a driver under drivers/. Scorers are code — scorers/{path-recall,anchors,verdict, hidden-suite,intact,matchers}.mjs, each { score(kase, view), aggregate(records) } in a registry; matchers (expected paths / regex / json-schema over the final text) makes an arbitrary prompt over an arbitrary case directory scorable. Mechanical scoring only, no LLM-as-judge. The eight roles become eight task files; the ripwire arms, two files differing in append, tools and env.

Subject — a model and its knobs, parsed by lib/subject.mjs:

--sweep: a model’s fixed knobs are its base; each sweep multiplies the base by its values, two sweeps cross, and a swept knob replaces the same fixed knob. --model A --sweep thinking=off,low --model B yields A:off, A:low, B. Every subject gets a canonical key, vertex-anthropic/claude-haiku-4-5:low@budget=1024,maxTokens=4000 (knobs sorted, absent ones omitted — the production subject is the bare ref); its directory name maps /→, :→, @→__, ,→+, keeps =, and is never parsed back (the record carries the key). A validator refuses a knob the model’s API or the task’s path cannot honour — before any call, never dropped; battery knobs REF prints the table, which is written from the spike (unit 1), not from memory.

Knobs on the wire (lib/knobs.mjs, one module, two paths):

On both paths lib/knobs.mjs writes request-knobs.json into the lane’s evidence — the streamSimple options passed, or the payload fields the handler set (reported back through the file named in BATTERY_KNOBS_OUT) — so a record shows the knob reached the request.

Stack — which extensions a lane runs under (lib/stack.mjs):

Single-call tasks ignore the stack (no session); their key records none. An agent-loop lane under a stack gets an agent dir seeded the way tools/consumer-sim.mjs seeds one (settings.json, trust.json, auth.json) plus what the battery adds: a gadhs-pi-modes.json overlay with the mode’s model pinned to the subject — pi-modes switches the session model on mode entry and would otherwise measure the wrong model. The stack key is the preset name or the sorted list; its fingerprint is each entry’s package version (content hash for a bare file). Cost under a stack includes the helpers it spawns: the lane folds the gadhs-subagent trace entries (#105) when the JSON event stream carries them; when it does not, the record says costIncomplete: true and the summary shows it.

Store — store/<task>/<stack>/<subjectKey>/<case>-<run>.json plus the .evidence/ directory beside each record as now (transcripts stay gitignored). A record keeps today’s shape (wallMs, usage, stopReason, score, turns) and gains subject, stack and a fingerprint:

A record is stale when its promptSha, caseSha, corpusPin or stack fingerprint differs from what the same permutation would use now. A model swapped upstream behind the same id is not detectable: pi-ai 0.84.3 declares responseModel on the message and populates it for no provider. The page prints measuredAt per cell and re-measurement is a deliberate --refresh; the record stores responseModel if it ever arrives.

summary.json is derived from the store, per permutation, with today’s aggregates; battery summarise rebuilds it from disk. Replacement: a slice already present is kept (today’s resume) unless --force, which deletes that slice’s records first — git history holds the old ones.

Manifest — battery.json:

battery fill runs every manifest permutation that is missing (with --stale, stale too), under the budget gate projected over the whole cross product. battery run runs the permutation on the command line whether or not the manifest lists it. The page renders the manifest’s cells only (operator’s choice; extras stay in summary.json): the matrix reads each role’s production subject (bare ref, bare stack) so defaults stay argued from production settings, other permutations of the task render in an experiments table beside it, a cell with no records reads not measured, a stale one carries its mark and what changed.

run.mjs becomes cli.mjs with one verb table; task.mjs keeps ONE battery verb that forwards to it, and battery-render is deleted (an alias for a tools/-only surface). --out goes: there is one store. --jobs, --budget and the fail-closed projection stay as they are.

Prompts are production’s, read at run time; scoring is mechanical; the coder sandbox (bwrap, SANDBOX_HIDDEN, repoUntouched), the corpora and their pins, the blind-authorship test, writeEvidence, the budget gate and inLanes are kept under the new names. Defaults remain starred and argued; a shipped default change is still its own issue citing the page.