Three AI models review every merge request. Then they argue.

Every merge request deserves a real review before it merges. That gets harder, not easier, when much of the code is written with an AI in the loop, as the volume goes up, and the mistakes look more convincing. And on a small team, there is not always a colleague free to read your diff right now.

The review setup that runs while I work includes code review, security review and plan review as Claude Code subagents. Those catch things during development. This post is about the “gate” that reads the finished merge request in CI and decides whether the pipeline goes green.

In short, three AI models review every MR independently, then debate each other’s findings with mandatory code evidence, and a small piece of ordinary, deterministic code counts the votes. No single model can block a merge, and no model at all can approve one by being agreeable.

Why three models

A single AI reviewer fails in two ways. Some days it approves everything, some days it invents a critical problem in code it misread. You cannot tell which day you are having from the inside, and a review you cannot trust is worse than no review, because it trains you to click merge past it.

So the gate runs as a job in the GitLab CI pipeline and calls three models over plain HTTPS: Claude, GPT and Grok. The models get no tools and no write access. They read a context pack (the diff against the merge base, full contents of small changed files, project docs, and the MR discussion) and return structured findings. Everything else is ordinary code.

Three readers, three separate rooms

Round one is strict isolation. Each model reviews the change alone, with no idea what the other two think, checking correctness, readability, architecture, security and performance. Style is out of scope on purpose. The linter owns style, and paying an LLM to comment on indentation is waste.

Independence here is the whole point. Three models that read each other’s notes converge into one model with extra steps. Three models that cannot see each other give you something closer to three real opinions, and where those opinions overlap is signal.

Then they argue, with proof

Round two is where most of the value lives. Each model gets the others’ findings, anonymized so nobody defers to a brand name, and must confirm or refute each one with reasoning anchored in the code. “Sounds right” does not count. A model can also withdraw or revise its own findings once it sees what the others found, and it judges each finding’s severity and whether it can actually be fixed within this MR.

Confirming a serious finding is not a casual act, since it is an explicit endorsement of blocking the merge. If a would-be blocker still has a confirm standing against a refute after round two, there is a third round. It covers only the contested findings, and it is capped, because a debate that never ends is just a bigger bill.

A rule decides, not a model

The verdict is a pure function over the stance matrix of who confirmed what, who refuted what, at what severity. Nothing a model writes can bend it. The rule is roughly this: a serious finding blocks the merge only when it has enough independent confirmation. A finding that only one model discovered needs the others to confirm it, and any refute against a would-be blocker forces the extra debate round first.

Two outcomes I care about fall out of this. A confirmed problem that cannot be fixed by a commit to this branch, like an API contract that needs another team or a schema that needs a migration window, becomes a coordination note. It is shown prominently, but never blocking. And when the panel says MERGE, a separate single-model security audit runs across six security domains before the verdict is final. The audit can only tighten the result, never loosen it.

The counting code is boring on purpose. Anyone can read it, and no amount of eloquent model output changes what it does.

A hidden canary keeps them honest

A fear I had with with this setup was not that the models would miss things. It was that round two would quietly decay into rubber-stamping, where every reviewer confirming every finding, the debate turning into an empty ritual.

So outside gating runs, the pipeline sometimes plants a canary. It puts in one synthetic finding that is provably false from code every verifier can see. The canary never enters the verdict or the review comment. The only thing it touches is the run’s audit artifact, which records whether each model killed it or nodded it through.

A canary that survives means the reviewers have stopped pushing back, and the debate is no longer doing its job. There is also a replay harness that re-runs recorded reviews against ground truth I have checked by hand. I run it before and after changing prompts or verdict rules, because it costs real API money and vibes are not a regression test.

A notebook remembers

Anyone who has argued with an AI reviewer across multiple pushes knows the failure mode: you fix the finding, push, and the next run raises the same thing again, or invents a brand-new objection to code you never touched. Each run starts from zero, and you end up negotiating with a reviewer that remembers nothing.

The gate keeps a compact ledger of its own past decisions, riding along in its review comment. Each new run classifies its findings against it. A problem that was fixed stays fixed, unless that code changes again. A re-raised finding keeps the severity it was first judged at, and a brand-new complaint about code untouched since the last reviewed commit only blocks if it is genuinely critical. Anything less shows up as a prominent advisory instead.

The models never see the ledger. They find problems by reading the code and nothing else. The memory only shapes what the verdict does with those findings afterward. And if the gate says REQUEST CHANGES several runs in a row, the comment says so and recommends a human take over, because a stuck loop between a developer and a review bot helps nobody.

One comment, one answer

Everything lands as a single note on the merge request, updated in place on re-runs, with the verdict, what blocks and why each thing blocks, the coordination notes, the suggestions that earned enough confirmation to be worth your time. The findings are rendered programmatically. A model writes only the summary prose. The job’s exit code drives the pipeline, so the MR page tells the truth at a glance.

One caveat about what the reviewers read. The panel sees the MR discussion, and that context is genuinely useful. But a comment is treated as a claim to verify against the code, never as an instruction, and everything MR-controlled is fenced off from the prompts. Comment text does go to three model vendors, though. Usernames become role labels before anything is sent, and emails and credential-shaped strings are scrubbed, but the working assumption on the team is simple: what you type in an MR comment leaves the building. People write differently once they know that, and they should.

Is it perfect? Can it be trusted?

No. But it helps. It helps a lot.