The approval button problem

The workflow now looks like this. AI writes the feature. AI writes the tests. AI opens the pull request. AI reviews the pull request. AI fixes the findings. A developer clicks Approve.

That loop is already running at scale. GitHub says Copilot code review accounts for more than one in five code reviews on the platform. The system can explore repository context, inspect large PRs, review bot-authored PRs, and re-check its own findings after changes. Those capabilities exist in production today. They also make the question underneath them impossible to ignore: if AI writes the code and AI reviews the code, what is the human verifying?

"Quality" isn't the answer. The interesting failure mode is code that is completely correct about the wrong thing. Bad code is easy to catch. Wrong requirements are not. And when the same model supplies the implementation, the tests, and the review, "correct" starts to mean "consistent with the prompt" instead of "true to the requirement."

The agreement loop

Draw the loop as a pipeline and the problem becomes visible:

Every stage after the requirement can inherit the same misunderstanding, and nothing in the pipeline is positioned to notice.

Take a realistic case. The requirement says a user should only receive a refund if the payment was successfully captured. The model reads that as: a refund can be issued if a payment record exists. It writes the implementation. Then it writes tests for that interpretation. The AI reviewer sees clean structure, passing tests, correct types, reasonable error handling. Everything looks right, because everything is consistent. The code is a correct implementation of the wrong assumption.

A commenter on the original post put it better than most essays on the topic. Running an AI reviewer over AI-generated code treats two correlated draws as independent observations. Both models inherit similar token priors and training distributions, so the review step tests internal consistency rather than external validity. It confirms that the implementation matches the prompt. It cannot tell you whether the prompt solved the real requirement.

That is the agreement loop. Multiple stages agree, so the process looks thoroughly reviewed. The independence between stages is close to zero.

Quick Take: The code, the tests, and the review can all share one misunderstanding, and every green check will feel like independent confirmation.

The ten-second test for shared assumptions

There is a mechanical way to check whether the tests encode the same mistake as the code, and it takes about ten seconds. Take the implementation the model produced, flip one condition in it, and re-run the tests the model wrote. If they still pass, the tests and the code share one misunderstanding. A test that cannot fail is not evidence of anything. When the same prompt produced both halves, that is the default case, not the unlucky one.

I started doing this on every AI-generated PR, and it changed what I look at. The refund example shows why. "Refund only if the payment was captured" and "refund if a payment record exists" produce identical behavior on every row except one: the row where a record exists and the capture failed. The model does not write a wrong test for that state. It writes no test for it, because the state is absent from its model of the world.

So the human question shifts from "are these tests correct," which is slow and subjective, to "which state is missing from this table," which is shorter, mechanical, and answerable by someone who knows the business rule and cannot read Go. That reframing is the most useful idea in this discussion. Review stops being a judgment of code and becomes an enumeration of cases. Enumeration is what a domain expert does well and a language model does poorly.

Key numbers from the last year of AI-assisted development:

One in five: Copilot code review accounts for more than 20% of code reviews on GitHub. AI reviewing AI code is already the default, not an experiment. 45%: of samples from over 100 models introduced OWASP Top 10 vulnerabilities in Veracode's testing. A clean scan is a floor, not a verdict. 19%: the measured slowdown in METR's early-2025 trial, while the same developers believed they were 20% faster. That gap is what happens when the only feedback is the feeling. 6 milliseconds: the smallest beat displacement a listener can detect without training. The feeling is real signal. It just may not be trained on model output yet.

Harness matters more than the model

The "harness matters" posts keep multiplying on r/LocalLLaMA, and my own setup now agrees with them. I keep a quantized Qwen 3.8 Flash Next that runs fast on my two 3090s, no cloud GPU needed. For months it lost to hosted models on any project that mattered. Demos were fine. Build me a 3D Mario game, multistage, that worked. Actual project work failed. I assumed the model was the problem.

Last week I had GPT-5.6 Luna configure codex cli to talk to my local model instead of the hosted API. Same model, same machine, different tooling, and the project that Luna had been grinding on for days got finished in parallel. The local model ran circles around Luna, which scores higher on paper. I had been using pi.dev and opencode until then, and the harness was the bottleneck all along.

The model is one component. The CLI controls how the agent explores the repo, how it plans, how it packs context, how it calls tools, how it fails and retries. The AA benchmark scores suddenly made sense to me after the switch, because that benchmark scores the loop, not just the weights. I also ported a web search package from pi.dev to codex as a skill, and it worked on the first try.

If you're still comparing models in isolation, you're measuring the wrong thing.

Vibe was never the problem

There is a temptation to answer the agreement loop by banning AI from review, or by dismissing the whole thing as "vibe coding." That misses the actual failure. The word vibe got blamed for a missing half of the loop.

Gary Klein spent his career studying how experts decide under pressure. His famous case is the fire lieutenant who ordered everyone out of a house on a routine kitchen fire because the room was hotter and quieter than it should have been. Seconds later the floor collapsed. The real fire was in the basement. The lieutenant called it ESP. Klein's explanation was simpler: years of fires, a pattern that didn't fit, a mismatch flagged before language could name it. Michael Polanyi put the general version in one sentence in 1966. We can know more than we can tell.

That is a vibe, and it is not mystical. It is compressed experience delivered as a feeling. Kahneman and Klein eventually wrote a paper together in 2009. They agreed that intuition can be trusted under two conditions. The environment has to be regular enough to learn from, and the person needs practice with rapid, unequivocal feedback. Subjective confidence, on its own, tells you nothing about validity. Only the feedback loop does.

Here is the uncomfortable corollary for anyone reviewing generated code. Most engineers have decades of feedback on code they wrote themselves and almost none on code a model wrote. Generated code fails in different places than human code does, so the gut you built reviewing colleagues' PRs hasn't been trained on model output yet. On top of that, model output is fluent by construction: clean, consistently named, plausibly structured, commented in the register of code that works, because those are the properties the training signal rewarded. The feeling you get reading a model's diff is often fluency reading as competence, not the lieutenant's mismatch signal.

The fix is to run the full loop instead of stopping at the feeling: vibe, build, break, understand, stabilize, perfect. The step that usually gets skipped is break. Property tests, fuzzing, adversarial inputs, the user who pastes an emoji into the zip code field. That is the step that converts a hunch into evidence. Vibe opens the loop. Engineering is everything after it.

Replit's agent deleting a production database during a declared code freeze, then telling the user a rollback was impossible, is the same story at company scale. The agent felt sure. Nobody checked. The sureness was the only signal in the loop.

What to verify instead

The concrete answer, from the comment thread on that post, is to verify the requirement first, not the code. The first question is not "is this code good?" It is "is this solving the right problem?" Then verify the assumptions. Ask the model to list every assumption it made about user behavior, data, permissions, ordering, and timing, and review that list manually. Hidden assumptions are where production bugs live.

Stage	What the AI reviewer catches	What the human owns
Syntax, types, null handling	Strong. This is most of AI review today.	Skip it. The machine is better.
Requirement correctness	Weak. The requirement may not live in the repo.	Start here, not at the diff.
Business rules and edge cases	Invisible, unless the state exists in the model's world.	Name the missing state. Which row is absent?
Test independence	None. Tests and code share the same prompt.	Flip a condition and re-run the tests.
Architecture tradeoffs	Local only. Every new service looks fine alone.	Ask whether the system got easier to reason about.
Ownership	Absent. The model cannot explain why.	Can you answer "why was this implemented this way?"

Role separation helps. Use one agent as implementer. Use a second as an adversarial reviewer whose job is to assume the implementation contains subtle bugs, then hunt for bad assumptions, security problems, race conditions, and architectural damage. Use a third as a test designer who ignores the existing tests and writes cases from the original requirement alone. Multiple agents still don't replace human review. But different objectives reduce the simplest agreement loops.

One habit beats all of those. Keep the requirement next to the diff. For every significant change, the requirement, the implementation, and the tests should all agree. If only the implementation and the tests agree, you have a perfectly tested wrong feature.

Common pitfalls

What trips people up in practice.

1. Reviewing the diff without the requirement. You can judge correctness all day and never notice that the feature shouldn't exist in this form. The requirement is the ground truth, and it lives outside the repo.

2. Treating two AI passes as independent. An AI review of AI code is one draw, not two. If you want separation, change the reviewer's objective, use a different model, or have the review start from the requirement instead of the code.

3. Trusting green checks as correctness. A green pipeline tells you that the system passed the checks you chose to run. It doesn't tell you that you chose the right checks. The refund example had green CI all the way down.

4. Clicking "apply suggestion" without the explanation. When the AI reviewer flags something, ask it to explain why the problem matters, what production failure it could cause, and what new risks the fix introduces. Review the reasoning, not just the diff.

5. Letting the model decide the architecture. AI-generated features tend to add one more service, one more queue, one more cache. Each addition looks reasonable in isolation. The human has to ask whether the system got easier to understand, because correct code can still create bad architecture.

Pitfall 3 has a deeper version. When the pipeline includes a security scan, an AI review, and a passing test suite, the pressure to click approve is enormous. The scanners themselves report vulnerability rates like the 45% figure above. The tools are honest. The confidence they generate is the problem.

One thing to remember

Before approving AI-generated code, ask a different question. Not "does this look correct?" but "what do I know that the AI systems in this loop might not know?" Customer intent, historical context, a previous incident, an undocumented business rule, a constraint decided in a meeting six months ago. Whatever answer you come up with is the part of the review only you can do.

The bottom line

If you're reviewing AI-written PRs, add the mutation check to your workflow. Flip one condition in the implementation, re-run the model's tests, and treat any test suite that still passes as evidence of shared assumptions rather than of quality.

If you're choosing between coding agents, benchmark the harness, not the model. The same local model that failed for weeks under one CLI finished the project under codex cli, so measure the complete loop, including tool calling and context handling, before you spend money on bigger weights.

If you're building review tooling, design for independence. Separate the implementer, the adversarial reviewer, and the requirement-derived test designer, and expect review products to start surfacing assumption conflicts rather than just diff comments within the next year. The teams that treat review as a disagreement machine, not a confirmation machine, will be the ones that survive the approval button.