The cheap model that gets to be wrong beats the expensive one that can't

When a bug is hard and spread across several files, the first instinct is to reach for the most expensive, smartest model and let it think it through in one shot, because surely the pricier one makes fewer mistakes. We believed exactly that, until we measured it. The numbers turned the belief inside out.

We put two models on the same work. The expensive one (Opus) got a single shot. The cheaper one (Sonnet) was allowed to loop up to three times. The bugs were made hard by mechanism, not by vagueness: two-stage bugs where the second only surfaces after you fix the first, bugs where the failing test points at the wrong module, and cross-file contracts where the easy fix turns another test red. Every one was proven to stay red under a shallow fix. Then we graded by running the whole test suite ourselves on every attempt, never trusting the agent's own report.

Part 1The result splits the room

On six hard bugs, here is what came out:

• Opus, one shot: 6/6, but 3.76x more expensive than the cheap model.
• Sonnet, one shot: only 4/6, at about 2.5x. More expensive, and it lost.
• Sonnet allowed three passes: 6/6, same as Opus, at just 1x cost. Half a million tokens against Opus's one-million-plus.

Put plainly, the cheaper model with the right to a do-over finished as much as the expensive one, at less than a third of the price. And on an easier round of ten tasks the loop was not even needed. Sonnet solved each on the first pass, while Opus still paid over 2.5x for the same output.

Part 2The number that settles it

If the story ended at "looping is cheaper," it would just be a good deal. But there is one figure that is stranger than that, and it is the whole point.

Sonnet on a single shot used more tokens than Sonnet looping three times (1.21 million against 0.49 million), yet scored worse (4/6 against 6/6). Same model, same tasks. The only difference was one thing: whether it was allowed to be wrong.

The version that could not miss spent more money and produced lower quality. That means looping is not merely cheaper, it is genuinely better and cheaper at once, a deal that should not exist under ordinary economics.

Part 3Why this happens

The reason the loop wins is not that looping makes the model smarter. It is exactly as dumb on every pass. What changes is its strategy.

Forced to be right on the first try, the model has no way to know whether its answer is correct until it submits. So it buys confidence through exhaustive in-head checking before it dares to answer. In the real experiment, one agent sat and read every one of the thirty-one modules the test imported, then compared the logic against what the test expected, line by line, by hand. It spent 248,000 tokens. Another spent 257,000 and was still wrong.

Give it three shots with a test suite telling it right from wrong, and the strategy flips instantly. It forms a quick hypothesis, ships it, and lets the truth judge. When it is wrong it knows immediately where, fixes, and fires again. 6/6 at forty percent of the tokens.

The principle fits in one line:

A checker to fire against is a license to be right cheaply.

The real cost of the order "do not miss" is the in-head verification, which is both more expensive and less trustworthy than running the actual test once. We were paying the model to guess whether it was right, when the machine could tell it for free, and more accurately.

Part 4How to use it

Three rules came out, and the last matters most.

1. For work that has a real pass-fail checker (a runnable test suite, numbers that reconcile, a linter), the default should be "cheaper model plus the right to N passes," not "upgrade to the expensive model." That saves roughly 4x on hard work and 2.5x on easy work, at equal quality.
2. Tell it plainly that it may iterate. That one sentence in the instructions flips the strategy from "verify in your head until sure" to "try it and measure," and that is the source of both the money and the correctness you save.
3. This is the line you must not cross. Work with no checker (writing, judgment, synthesis) must never use this rule. Looping with nothing to tell you right from wrong is just producing more garbage each round. We proved the reverse in another experiment: a small model looping without a checker lost badly. It is the strong checker that turns looping into an advantage, not the looping itself.

So looping is not a spell you sprinkle on to make things better. It is a mechanism that works when there is a truth judging at the far end. With no measurable finish line, running the loop again is just running in circles.

Next time you reach for the most expensive model to crack a hard bug, ask one question first: does this task have a checker that can tell right from wrong? If it does, let the cheaper model take three swings, and measure. There is a good chance you get the same work for several times less. Expensive does not mean worth it. Sometimes the most worthwhile thing is to let it be wrong once or twice, and let the truth do the teaching.

More in this series: which Claude model for which job · a test that never failed proves nothing