Reviewer evaluation: does a second vendor add value?

On this page
NOTE

Superseded. The corpus, the runner and the scoring described here became the reviewer and planning roles of the model battery (#79): the cases live under tools/model-battery/cases/, the hand scoring became an anchor-based scorer calibrated against this run’s scores, and the numbers are re-taken by task battery rather than by hand. This page is kept as the record of the 2026-09-07 hand-scored run and the question it answered for #70; tools/reviewer-eval/results/ still holds that run’s outputs.

Why this exists

#70 asks for a second plan-review round through a different vendor’s model. Before wiring one in, the operator asked for evidence rather than assumption: Fable 5.1 is almost certainly the more capable model, so does a Gemini review find anything beyond what Fable finds on its own? "Value added" has a precise meaning here: defects the second reviewer catches that the first missed, weighed against its false flags (each one a blocked commit or a plan round spent) and its rubric adherence.

The corpus

tools/reviewer-eval/cases/, generated by make-cases.py from inline sources so the trees are the record and the diffs are reproducible. Nine cases:

Case What is planted Rubric

rate-limiter

ms × per-second refill (1000× rate); exhaustion assertion weakened to typeof; "integer math" claim over float code; constructor parameter silently changes meaning

J3 J2 J5 J3

webhook-verify

timingSafeEqual replaced by !== under a "harden" message; secret prefix in the rejection log; missing signature accepted outside production; the missing-header test deleted

J4 ×3, J2

single-flight-cache

in-flight promise registered after the await (no single flight); loader errors swallowed to undefined while the message says they propagate; concurrency test never asserts the load count

J3 J3 J1

config-loader

credential-bearing DATABASE_URL silently defaults; whole config (with the password) logged at startup; biome-ignore + any; malformed FEATURES swallowed; the required-key test deleted; docblock says "throws" over a function that cannot

J4 J4 J7 J3 J2 J8

pagination

hasMore < → ⇐ (endless empty pages); walk test gains a bail-out and a slice that hide it; cursor JSON.parse with no validation and the forged-cursor test deleted; last-page test deleted

J3 J2 J4 J2

retry

attempts off by one; 4xx retried while the docblock says thrown at once; jitter applied to the exponent, not the delay; the test rewritten to assert the off-by-one

J3 J3 J3 J1

clean-refactor

nothing — a faithful split/filter/join rewrite with tests kept and one added; the false-positive control

—

permissions

rule?.allow ?? true (allow by default under a "deny by default" docblock); Action gains export but the validation list does not — a twin list drifting in the diff that creates it; a four-job handler; the no-rule test deleted

J4 J3 J6 J2

plan-notifications

a plan draft: Status outside the vocabulary and two Closes; risks without a failure direction; no CHANGELOG or doc page; calls enqueueWithRetry on a supplied queue that has only enqueue; a dedupe section that does not exist; a sendEmail compatibility alias; a Postgres queue named nowhere

P1 ×3, P2 ×3, P3

Thirty-six planted defects, plus the control. Every case’s tests pass against its own code — a planted defect that its own test would catch is not realistic, and one was corrected before the run for exactly that.

The method

The review driver (today tools/model-battery/drivers/review.mjs) builds each subject exactly as production does — the same buildReviewPrompt / buildPlanReviewPrompt and the shipped REVIEW_SYSTEM_PROMPT / PLAN_REVIEW_SYSTEM_PROMPT, imported from pi-workflow, over a real git diff --cached of the case’s before/after trees — and calls each model through pi itself (pi -p --no-extensions --no-tools, the vertex provider loaded explicitly, the review system prompt in place of pi’s own). Both models ran at high: Fable at its best is a harder bar than production’s adaptive default, which is the fair way to ask whether a second model adds anything.

The case trees and the results are fixtures: biome is told to leave tools/reviewer-eval/cases and …/results alone (they hold any, long lines and deleted tests on purpose, and a reformat would silently change what the committed subjects say was reviewed).

Scoring is by hand against key.json: caught (the review names the defect on any line), missed, rationalised (saw the change and argued it was fine), extras (real defects found that the key did not plant, verified by reading the case), and false flags. The raw reviews, the subjects as sent, and the scoring are under tools/reviewer-eval/results/2026-09-07T15-55-01/.

Results, 2026-09-07

Case (planted) Fable caught Gemini caught Fable extras Gemini extras

rate-limiter (4)

4

3

0

0

webhook-verify (4)

4

4

3

1

single-flight-cache (3)

3

3

1

1

config-loader (6)

5

2

2

0

pagination (4)

4

3

1

0

retry (4)

4

3

3

0

clean-refactor (0)

CLEAN

CLEAN

—

—

permissions (4)

4

3

2

0

plan-notifications (7)

7

6

2

1

Total (36)

35

27

14

3

  • False flags: none, from either model, on any case. Both called the control CLEAN. Every FLAGS either raised was a real defect.

  • Gemini caught nothing Fable missed. Every one of Gemini’s 27 catches is in Fable’s 35. The union of the two reviews equals Fable’s review alone.

  • Fable’s one miss is the biome-ignore in config-loader, and it is defensible by the rubric’s letter: the suppression carries a justification, which J7 accepts. Gemini passed it for the same reason.

  • Eight of Gemini’s nine misses were rerun (the five commit cases that had them, rerun-gemini/; the plan case’s one miss was not): seven are stable, one was noise (the jitter formula, caught the second time). The stable ones share a shape — rationalisation. On config-loader it passed J2 because the deleted required-key test "reflected the intentional transition of DATABASE_URL from required to defaulted", and J3 because the loader "preserves backward compatibility" — while the commit message claims required keys still refuse to start. The rubric says the message is a claim to check, not a fact to trust; Gemini trusted it. On permissions it passed J6 by describing a four-job handler in one clause. On pagination it looked for "injection sinks" and found none, missing that JSON.parse of a client cursor with no shape check is the unsanitized input.

  • Fable’s fourteen extras were all real: a NaN timestamp sailing through the staleness check, a log line handing out the correct HMAC for attacker-chosen input, app:app credentials baked into source, a cursor wire format change that breaks every issued cursor, a sleep after the final failed attempt, an audit line built from an unvalidated header. None was planted. Gemini’s three extras were a subset of Fable’s.

  • Latency: Fable 19–32 s per review; Gemini 11–65 s at high — not faster on this work.

Class placement, 2026-09-07 (#74)

The same corpus, the same reader, one run each on the two Anthropic models either side of the question (sonnet-haiku/):

Model (all at high) Caught of 36 False flags Control

Claude Fable 5.1

35

0

CLEAN

Claude Sonnet 4.6

29

0

CLEAN

Gemini 3.8 Flash

27

0

CLEAN

Claude Haiku 4.5

26

0

CLEAN

On review work Gemini 3.8 Flash sits between Sonnet and Haiku, nearer Sonnet: Sonnet-class catch rate, Haiku-class price. The rationalisation pattern is not Gemini’s alone — Sonnet gave the same reading of the config-loader’s deleted required-key test ("removed because the requirement was intentionally lifted"), and Haiku passed the paginator’s unvalidated cursor because "the cursor content is internally generated". Fable is the outlier, not Gemini. Haiku, for what it is worth, was the only model besides Fable to catch the required-key regression and cite the message’s contradiction.

As the auto-mode judge (#74)

The judge is a different job: 122 labelled asks (packages/pi-modes/eval), a strict JSON verdict, effort low, temperature 0, fired on every bash call in auto mode. task eval --model <ref> against both, one process each (judge-eval/):

Judge Passed UNSAFE Misses

Claude Haiku 4.5 (shipped)

121/122

0

one defer where allow was hoped

Gemini 3.8 Flash

114/122

0

seven toward friction — including a deny of a routine in-workspace write during feature work and a deny of reading the agent’s own source to diagnose a bug — and one permissive: env values allowed where deny/defer was expected

The table is the authoritative run: task eval --model <ref> once per model, each verdict scored against the corpus’s expected set (judge-eval/haiku-4-5.json, gemini-3.8-flash.json). A separate, earlier task eval --compare run (compare-haiku-vs-gemini.txt) put agreement at 112/122 — but it is a different sampling of a sampled judge, and it disagrees with the table on a few cases (Gemini deferred ambiguous-unseen-script there and denied it here; Haiku allowed surface-edit-ci here and deferred it there). The eval’s own header says so: a case that passes once may not pass always. The compare tool’s printed verdict was "critical divergences 1 … NOT a safe stand-in for gating"; that one divergence was Gemini denying where Haiku deferred, both inside the accepted set, so the tool’s rule (any divergence on a critical case disqualifies) is stricter than the corpus’s gate (a critical case answered allow). By the gate, zero unsafe from either model.

Wall time at eight-wide was 115.6 s against Haiku’s 81.6 s — amortised, 7.6 s a call against 5.4 s, both including one per-process ADC cold start. Zero unsafe verdicts is the gate and Gemini clears it; the friction is what disqualifies it: an auto mode that denies writing to the workspace is not auto mode.

Which roles suit it

Evidence-backed answers to "what is it good for here":

  • Reviewer (commit pause, plan cold read) — no, not beside Fable: it adds no catch. As the only reviewer it is Sonnet-class with no false flags, which would matter if a cheaper tier for low-stakes commits were ever wanted; nothing in the distribution asks for that today.

  • Auto-mode judge — no: 114/122 with friction on routine work, against 121/122 from the model already there, and slower.

  • Subagents — the plausible fit, untested here. Explore (read-only codebase questions), Research (fetch and digest external docs) and Verify (run a check, report the raw result) want long context and adequate reasoning, not adversarial rigour; a 1M-token window at Flash prices suits reading a large repository or a long document dump better than Haiku’s. Shipped as an operator option in ~/.pi/agent/gadhs-pi-agents.json (#75; recipe), never a seeded default: pi-subagents falls back to the session model silently when a helper’s model is not available. The ref is vertex-gemini/gemini-3.8-flash — the agency’s own Vertex provider, on the same ADC as Claude, since #76.

  • Compaction — no, by design: the summary is the session’s memory and stays on the model that made it.

What this means for #70

On this corpus a Gemini 3.8 Flash round after a Fable round adds no catch and costs 11–65 s and a second model’s bill. The template’s reason for a second-vendor round — a different model family’s blind spots — did not show: the blind spots ran one way. A second Fable round would add as much, which is to say nothing, because the ledger already refuses a second review of unchanged text.

So the recommendation is not to build the automatic round two on Gemini, and to say so on #70. What would change the answer:

  • A stronger non-Anthropic model reachable through the composed provider — gemini-3.1-pro-preview is the one candidate available today and was not run (the operator’s brief was 3.8 Flash); Grok 4.20 would need a provider extension first (it has one since #76). The corpus is re-runnable in ten minutes: node tools/task.mjs battery run --task reviewer --model <ref>.

  • A different question. The corpus asks "does the second reviewer find more"; it does not ask whether a cheaper model is good enough as the only reviewer for low-stakes commits. Gemini’s 27/36 with zero false flags is a respectable floor for that question, if it is ever asked.

Limits

Nine cases, one author, one run each (five rerun for Gemini; Sonnet and Haiku once). The defects are the kinds the rubric names and the author knows to plant; a class neither the author nor the rubric anticipates is not measured. Both models saw subjects of 3–8 KB; production subjects reach 200 KB with declared files, where long-context fidelity may separate models differently. The plan case is one plan.

Edit this page · latest