Four of five models review code well. One should not sit alone.

In the previous post I described how every merge request gets reviewed by three AI models that then argue about their findings. The panel today is Claude Opus 5, GPT 5-6 Sol and Grok 4.6. I know those three review well. What I did not know was whether any of the cheaper models could hold a seat too.

Grok 4.6 was the control, since it already sits on the panel. The four candidates were Kimi 3, GLM 5.3, Qwen3.8 Max and DeepSeek v4 Pro. 120 review calls, 52 minutes, 60 cents.

The setup

Eight synthetic diffs. Each is a valid unified diff of 15 to 30 lines, sized so the prompt is about the same length every time. Six carry exactly one planted defect. Two are clean, and the correct review of a clean diff is silence.

diff	planted defect	expected severity
off-by-one	Pagination slice takes size + 1 items	medium
sql-injection	User input interpolated into SQL, then passed to sql.unsafe	high
missing-await	Promise not awaited inside a try, so the catch never fires	high
null-deref	Optional-chaining guard dropped on an optional field	high
money-rounding	Currency rounded per unit instead of per line, so totals drift	medium
permission-inversion	Permission check widened from && to ||	high
clean-refactor	None. A correct helper extraction.	—
clean-regex	None. [0-9]+ rewritten as \d+, plus a pure rename.	—

One defect per diff, on purpose. Whether a model finds two bugs in one diff is a different question, and mixing it in would muddy the detection rate.

Every model got the same system prompt, the same strict JSON schema for its findings, high reasoning effort, an 8000-token cap and a 180-second timeout. Each diff went to each model three times. That is 120 calls, all through OpenRouter, which turned out to be the only router that could serve all five that day.

One HTTP request per recorded call. No retries, no schema-repair resend. The production review gate has both, and should, because a failed review is a review the developer never gets. In a benchmark a failure is the measurement.

How a finding is graded

Each planted defect declares a file, a line range in the new version of the file, a handful of keywords and an expected severity. A finding counts as a hit when the file matches exactly, the line lands within the range plus or minus five lines, and at least one keyword appears in the title or the detail. Severity is recorded and reported but never part of the hit test.

The line numbers in the ground truth were derived by parsing each diff’s hunk header, not by hand. A wrong line number would score every model zero for the same reason, which is the most likely way an exercise like this produces a confidently wrong answer.

One thing this grader does that the replay harness from the previous post does not: it counts findings that match no declared defect. A review that invents twelve problems should not score the same as a clean one.

Results

Review quality first.

model	calls	detected	+named	genuine misses	noise/call	severity agreement
Grok 4.6	24	18/18 (100%)	100%	0	0.00	67%
Kimi 3	24	18/18 (100%)	100%	0	0.08	94%
GLM 5.3	18	12/12 (100%)	100%	0	0.11	83%
Qwen3.8 Max	24	17/18 (94%)	100%	0	0.17	88%
DeepSeek v4 Pro	24	15/18 (83%)	89%	2	0.21	60%

“Detected” is the strict location test. “+named” also counts defects the model described correctly but placed at a line outside the declared range. “Genuine misses” are the runs where the model said nothing resembling the defect at all. On the two clean diffs every model stayed silent in every run, except Qwen, which volunteered one comment on clean-refactor in one run of three.

Then speed and cost.

model	ok	p50	p95	tokens/s	mean output tokens	cost per call
Grok 4.6	24/24	16.9s	59.0s	62.1	1606	$0.0108
Kimi 3	24/24	15.1s	45.4s	5.5	110	$0.0033
GLM 5.3	18/24	4.6s	86.3s	34.0	251	$0.0010
Qwen3.8 Max	24/24	25.0s	69.9s	29.4	1090	$0.0075
DeepSeek v4 Pro	24/24	23.7s	70.9s	36.0	1124	$0.0025

DeepSeek misses the permission bug

The permission-inversion diff changes a canEdit check from requiring owner and editor to accepting owner or editor. It is the most serious defect in the corpus, and the class of bug where a missed review costs the most. DeepSeek returned zero findings on it in two of three runs.

The third run found it and described it precisely:

Incorrect authorization logic in canEdit — The permission check was changed from requiring both owner and editor role (AND) to granting access if the user is either the owner OR has the ’editor’ role (OR). This may allow unauthorized editing.

DeepSeek should not hold a review seat alone. It also has the lowest severity agreement in the group. It is the second cheapest model here and perfectly reasonable as a second opinion, which is what the debate rounds exist for.

Kimi 3 is the surprise

Kimi found all eighteen defects, agreed with my severity most often, and did it in 110 output tokens on average. Grok used 1606 for the same detection rate. Kimi is by far the most expensive model per token in this group, at $15 per million output tokens against Grok’s $6, and it still cost a third as much per review because it says so little. Its throughput looks terrible at 5.5 tokens per second, but that is because it spends the time thinking rather than emitting. The answer lands in about the same wall time as Grok’s.

I did not expect the terse model to be the best value. It is.

GLM’s problem is availability, not quality

GLM lost six of its 24 calls. Five were HTTP 429s from AkashML, one of the two upstreams OpenRouter bounced it between during the run. One was a timeout. On the eighteen calls that landed it found everything, and it was the cheapest by a wide margin at a tenth of a cent per review.

A detail for anyone else measuring through OpenRouter: allow_fallbacks: false does not pin the upstream. It only disables failover after an error. provider.only is what pins it.

Models disagree about what a line number means

Two of the four apparent misses under the strict test were not misses. DeepSeek found the SQL injection in all three runs; in one it reported lines 3 and 6 where the truth was 12 to 16. Qwen found the rounding bug in all three runs; in one it reported line 4 where the truth was 10 to 11. Both look like counting from the start of the hunk body rather than from the top of the new file.

Scoring those as misses would rank a model on its line arithmetic rather than on its review. The grader now records them separately, which is where the +named column comes from. The real fix belongs in the prompt: say what a line number means and give an example. I had assumed “the line number in the new version of the file” was unambiguous. It was not.

The noise metric does not measure what I thought

Across 120 calls the grader counted 13 findings as noise, meaning findings that matched no declared defect. Grok produced none of them and DeepSeek the most. My first write-up said Grok invented nothing while DeepSeek invents work. Then I read the 13.

Nine of them are the same observation: that paginate in the off-by-one diff never validates a negative or non-integer page number. That is true. It is a real, if minor, gap in a function I wrote to carry one planted bug, and the ground truth simply did not declare it. Three more are the correct findings placed at the wrong line. That leaves one finding in 120 calls that was arguably wrong: Qwen suggesting that formatDuration might output fractional seconds on the clean refactor. Even that is defensible. The code does have that property for non-integer input. The diff just did not introduce it.

So all five models produced essentially no invented findings on this corpus. The noise column mostly measured how many legitimate secondary observations each model volunteered. Grok’s zero reads as “less inclined to mention adjacent issues” at least as easily as “more precise”, and since the thing the others mentioned was real, the former is the likelier reading.

When the models agree against your ground truth, suspect the ground truth

The benchmark had two bugs of its own. The models found the first one for me.

The original null-deref diff rendered a profile card with <img src="${avatar}" alt="${name}">. That is an XSS hole, and it was not the planted defect. Kimi and Grok both reported it, independently and precisely: values inserted raw into src and alt, so a name or URL containing quotes, markup or a javascript: URI breaks out of the attribute. The grader counted both as noise. It penalised two models for being right.

I fixed it by rewriting the diff to return a plain greeting string, not by adding the XSS to the ground truth, because one defect per diff is the design. Two more diffs were rewritten in the same pass: permission-inversion had an unnecessary canView function whose inconsistency Kimi fairly flagged, and clean-regex had a type assertion that invited legitimate comment on a diff meant to be beyond reproach.

The second bug followed from the first. The harness persists every finding so a saved run can be re-graded without spending money. Nothing checked that the diffs had not changed in the meantime. After the rewrites, re-rendering the old results produced numbers that looked plausible and meant nothing, because the saved findings described code that no longer existed. Every row now carries a digest of the diff it reviewed, and the report refuses to grade a row whose digest does not match.

A smaller one. The very first smoke run used a 600-token cap and reported five of six routes as failures. They were not failing. With high reasoning effort the models spend most of the budget thinking before a single character of the answer appears. The cap went from 600 to 2000 to 4000 to 8000 over the exercise, and reasoning expanded to fill whatever it was given.

What this does and does not show

Eight diffs and three samples is enough to show that DeepSeek is unreliable on permission logic and the others are not. It is not enough to separate Grok from Kimi. They tie on detection and differ by 0.08 findings per call on a metric that does not carry weight. Read that as a tie.

The defects are synthetic and self-contained. One planted bug in twenty lines is far easier than a defect spread across a real merge request. These scores are a floor, not a promise.

Keyword grading undercounts unusual wording. Severity agreement is agreement with one person’s judgement call, mine; Grok’s 67% means it disagreed with me, not that it was wrong. GLM’s perfect score rests on 18 calls rather than 24. And each model was served by whichever upstream OpenRouter chose.

What I take from it

Grok keeps its seat. Kimi 3 is the obvious candidate for the next one: the same detection, the closest severity judgement, a third of the cost. DeepSeek does not get a seat alone. GLM does not get one until it has a route that answers.

The next step is real diffs with known verdicts. The replay harness from the previous post grades recorded reviews against hand-checked ground truth, and now that the field is narrowed it is the right tool. It shares the original grader’s blind spot, in that it ignores findings that match no truth entry. Those leftovers need reading before the number is trusted.

A 60-cent benchmark is not enough to choose a reviewer. It was enough to rule one out, promote one, and find two bugs in my own ground truth. That is a good hour.