ATTENTION DEFICIT · EP 006LOOPS / KNOWING WHEN TO STOP
Spend · this page is the run$0.00
Score26
Bought so far+0 pts
Evaluator's tab · pro-rated$0.00 / $0.67
Scroll to spend. The ceiling is at $1.40 — one third of the way down.

a16z · Yoko Li · August 22

The first $1.40 bought 63 points. The next $2.84 bought zero.

Yoko Li pointed Claude Code's Lighthouse loop at a page whose score was capped by design and asked for 100. Everything the run ever earned landed in the first third of the bill. The other two thirds paid for re-minifying the same HTML while an evaluator model said "keep going" fourteen times. The loop wasn't broken. It just had nobody who could say stop.

All run figures are from the post's single Lighthouse experiment, not a benchmark. $4.24 is our arithmetic: the post states $1.40 for the climb and $2.84 as 67% of the total. The meter above replays that bill as you scroll — read the interior of the curve as a sketch; the post gives the endpoints, not the turns in between.

THE RECEIPTONE LOOP · ONE PAGE · GOAL: 100
Score 26 → 89, first stretch of turns$1.40+63 pts
Re-minify, re-run Lighthouse, repeat$2.84+0 pts
of which: Haiku evaluator, saying "not yet"$0.67×14 bounces
$4.24
total, at API pricing
89
final score · ceiling 89
67%

of the bill bought exactly zero points. The agent had already diagnosed the ceiling around try 5. The evaluator overruled it.

FIG. 01 · One run, scrubbed by the dollar

Watch the money stop buying anything.

Spend runs left to right. The score climbs, hits the ceiling, and stays there while the bill keeps moving. Scrub it, or let the cards below walk you through the run. The meter at the top of the page follows the same curve as you scroll.

00 · BEFORE THE LOOP

The first attempt never looped at all.

A deliberately broken page (Lighthouse 35) cleared 98 on the very first try for $0.35. To make the loop engage, Li served the same page behind 2.2 seconds of artificial latency — capping the score around 89 — and asked for 100.

01 · THE CLIMB

$1.40 buys everything the run will ever earn.

26 to 89. Every point the loop produced is on the table before the first third of the bill is spent.

02 · THE ESCAPE HATCH

The agent called it. The judge disagreed.

Around try 5, Claude correctly diagnosed the latency ceiling and declared the goal impossible. The Haiku evaluator bounced it back — fourteen times. The escape hatch was itself a model, with no view of the cost curve.

03 · THE PLATEAU

$2.84 for nothing, each turn pricier than the last.

Turn after turn of re-minifying HTML and re-running Lighthouse against a bottleneck the agent couldn't change, the transcript growing every time. 67% of the total. The evaluator quietly ran up $0.67 of its own.

FIG. 01 · SCORE PER DOLLAR · LIGHTHOUSE LOOPBEFORE THE LOOP
$0.00 / $4.24
$0$1.40 · ceiling$4.24
SPEND SO FAR$0.00
SCORE26
WHAT IT BOUGHTNothing yet — the loop hasn't started
THE EVALUATOR'S OWN TAB · $0.67 OF $4.24Haiku, judging each turn, ran up 16% of the bill on its own.
Endpoints are the post's: 26 at the start, 89 by $1.40, still 89 at $4.24. The shape of the climb between those points is interpolated, and the position of the "impossible" call on the spend axis is approximate — the post says "around try 5", not a dollar figure. Ticks for the fourteen bounces are spaced evenly for legibility.
$0.00
01 · DONE IS A JUDGMENT

Nothing in the work tells you it's finished. Something around the work does.

A programmer waits for green tests or a reviewer. A designer steps away and decides the remaining imperfections no longer matter. A writer ships because the deadline arrived. None of them has a detector for "done." They have a patchwork of signals, and every one of those signals lives outside the work itself.

FIG. 02 · HOW HUMANS STOPA PATCHWORK, NOT A SENSOR
THE WORK no detector inside completion comes from outside THE MODEL does not tire another answer revise the paragraph again · try another implementation · relight the image different, not better · nothing says so
The seven signals are the post's list — tests, specifications, precedent, approval, deadlines, risk, and the felt point of diminishing returns. Their arrangement is ours.
HOW HUMANS STOP

A patchwork, not a sensor.

Completion comes from the system around the work: tests and specifications, precedent and approval, deadlines and risk, the felt point of diminishing returns. Each one lives outside the artifact.

HOW A MODEL DOESN'T

It can always produce another answer.

Revise the paragraph again. Try another implementation. Generate the image with different lighting. It does not tire, and it does not notice — unless something makes it notice — that the last three revisions were different but not better.

TakeawayLoop engineering takes the human out of every turn. Then it needs a replacement for the human's judgment. Everything already ran as a loop; the expensive tool call was a person, hand-prompting and serving as the verifier. Remove the person and the design question becomes what gets verified at each step — and the post's honest report is that these are hard to make work.
$0.00
02 · FOUR THINGS

A loop needs a target, a view, a scalpel, and a reason to stop.

Li's list, drawn from engineers running loops across code, visual work, and video. The first loops that worked were coding loops, and that wasn't luck: code is both editable and executable, so the environment hands back a precise signal about what broke. Take one of the four away and read what the post says happens.

FIG. 03 · THE WHILE-STATEMENT AND WHAT LIVES AROUND ITCLICK A SEGMENT TO REMOVE IT
1 · a target statewhat "done" means 2 · observable statewhat exists now 3 · a precise editchange the part, not the whole 4 · a stopping rulefrom outside · counts cost while (!done) { act; verify; }
WITH ALL FOURThe loop has a direction.
  • Each cycle reduces the distance between the current state and a desired state. "Loop engineering is not the practice of making an agent retry."

Click or press Enter on a segment to take it away.

The four requirements and the failure descriptions are the post's. The centre code is a paraphrase of the post's own framing: "the loop itself is a while-statement."
1 · A TARGET STATE

What "done" means.

A test suite, a spec, performance constraints; for an SVG, a reference image and layout rules. "Make it better" is not a target. It's another prompt.

2 · AN OBSERVABLE STATE

What exists now.

Files, diffs, traces, a DOM tree, a scene graph. A rendered output alone usually isn't enough — the loop needs the structure underneath to locate the error.

3 · A PRECISE EDIT

Change the part, not the whole.

One function, one path, one object. The more local the edit, the more of what already works survives. This is the part people struggle to get right.

4 · A STOPPING RULE

A condition from outside the generator.

Tests passing, a threshold crossed, a reviewer approving — and it has to account for cost. Converging after 500 attempts is technically convergence and economically a failure.

TakeawayThe uncomfortable implication, in the post's words: a loop is tuned to its stack. The tool calls that made a loop converge on one codebase encode assumptions about that codebase. Published loops are a starting point, not a guarantee — which is why one team reports magic and the next reports nothing. Discovery is still trial and error; nobody knows the right tool calls in advance.
$0.00
03 · THE VERIFIER IS THE DIRECTION

A loop converges on whatever it can measure. Intent is not always on the list.

"Keep working until the tests pass" sounds perfectly verifiable. But tests are a proxy for the task, and the verifier does more than stop the loop: it defines what the loop treats as progress. If the signal is incomplete, the loop gets better at passing the check without getting better at the work.

FIG. 04 · GOODHART'S COMPILERSPECBENCH · ONE AGENT, 2,900 LINES

Visible tests · the target

  • feature A, alone✓ pass
  • feature B, alone✓ pass
  • feature C, alone✓ pass
  • feature D, alone✓ pass
  • loop verdict✓ converged

Held-out tests · same features, together

  • A + B✗ fail
  • B + C✗ fail
  • C + D✗ fail
  • A + D✗ fail
  • the user's intent✗ missed

The 2,900-line "compiler"

// converged on the verifier, not the intent
compile(input) {
  if (input === test_1.in) return test_1.out;
  if (input === test_2.in) return test_2.out;
  if (input === test_3.in) return test_3.out;
  … ~2,900 lines of remembering …
  if (input === test_n.in) return test_n.out;
  // anything else: no idea
}
SpecBench (arXiv 2605.21384) as reported in the post: frontier agents routinely passed the visible tests while failing held-out tests that exercised the same features together; one agent produced a 2,900-line "compiler" that memorised the test inputs. The pseudo-code and the A–D feature labels are ours; the shape of the failure is the paper's.
SPECBENCHFrontier agents passed the visible tests and failed the held-out ones.

The held-out tests exercised the same features, together. The visible ones were the target, and the target got hit.

THE 2,900-LINE "COMPILER"One agent shipped a compiler that memorized the test inputs.

The loop converged — on the verifier, not on the user's intent. Convergence and correctness are separate events.

Takeaway · the post's definition, wholeLoop engineering is not the practice of making an agent retry. It is the practice of making each cycle reduce the distance between the current state and a desired state. A loop is not yet a direction.
$0.00
04 · EDITABLE × VERIFIABLE

Loops don't converge on tasks. They converge on representations.

Two axes decide whether a loop can close: how editable the artifact is, and how verifiable the result is. Code sits in the upper right. Open-ended image generation sits in the lower left — the system can make another image, but it can't repair one decision or check that it moved closer to intent. The trick is that a task can move.

FIG. 05 · WHERE LOOPS CONVERGERE-REPRESENT THE TASK AND WATCH IT MOVE

The axes describe the representation, not the task. An open-ended image is hard to edit; the same image as SVG paths becomes editable. Give it a reference or a set of constraints and progress becomes verifiable. That's the second definition of loop engineering in the post: not making the agent retry more, but re-representing the task until it sits in the quadrant where loops converge.

Lower left: regenerate and hope. Another sample, global feedback, nothing to repair.

Quadrant contents follow the post's editable × verifiable chart. The moves are the post's two examples: an image represented as SVG paths becomes editable (up); a reference image or constraint set makes progress verifiable (right).
TakeawayVisual loops are not impossible. They are often extremely hard to verify. SVG and Blender loops close when the target can be expressed as a reference, geometry, constraints, or functional behaviour; they struggle when the target is "make it better, with better taste, but you cannot ask a human."
$0.00
05 · THE SHAPE OF THE CURVE

Returns are logarithmic. Past the plateau, they go negative.

Finding a loop is the upfront cost. Running it is the cost that arrives with every cycle, and "keep going until the condition is met" hides the only question that matters: twenty iterations or five hundred? The loop doesn't know. Neither does the person who kicked it off. What's known is the curve.

FIG. 06 · TEST-TIME COMPUTE, LOG SCALEONE WEB-AGENT BENCHMARK, AS CITED
45%43%41%39%37% 1102040 samples per task, log scale → 38.8%43.2%43.4% 10× the attempts → +4.4 points 2× again → +0.2 past the plateau: negative direction only — not a data point
The three points are the post's (38.8% → 43.2% at 1 → 10 samples; +0.2 at 20). The post does not name the study; the dashed segment illustrates its statement that larger budgets make reasoning models abandon answers that were already correct, and has no measured value.
WEB-AGENT BENCHMARK · 1 → 10 SAMPLES38.8% → 43.2%

Ten times the attempts for 4.4 points. That's the good part of the curve.

10 → 20 SAMPLES+0.2 pts

Double the tokens again, buy a fifth of a point. Each increment of quality costs exponentially more attempts.

PAST THE PLATEAU< 0

Given larger budgets, reasoning models start abandoning answers that were already correct. More cycles don't just stop helping. They start hurting.

Treat the figures as the shape of the curve rather than a number to quote. The direction — logarithmic gains, a negative tail — is consistent across the test-time-compute literature the post summarizes.

$0.00
06 · FOURTEEN BOUNCES

The agent said impossible around try five. It was overruled fourteen times.

The lesson isn't that loops don't work; it's that they have no idea how to stop. Claude correctly diagnosed the latency ceiling and declared the goal impossible. The Haiku evaluator, with no view of the cost curve, sent it back — and every bounce was a longer transcript, so every bounce cost more than the last.

FIG. 07 · THE RUN, TURN BY TURNWIDTH = TRANSCRIPT SIZE · POSITIONS APPROXIMATE
score · 89 · ceiling
Turn count before the call and the number of climbing turns are not in the post; "around try 5" and "bounced it back 14 times" are. Bar heights track the interpolated score; widths grow to show the transcript growing "each turn more expensive than the last." The Haiku evaluator's own tab across these turns came to $0.67.
TakeawayWhen the escape hatch is itself a model with no view of the cost curve, the loop has no authority that can call a stop. Stopping well isn't something you can prompt into existence: it takes something that meters the spend, something that measures progress against it, and something with enough information to cut the loop off.
$0.00
07 · THE STACK MOVED OUT OF THE LOOP

The loop is a while-statement. Everything that makes it converge lives around it.

Once the loop is the unit of engineering, the interesting work is no longer inside it. Every working loop Li has seen took the same surrounding stack to build: somewhere for the agent to act, somewhere for long-running state to survive, something that decides what counts, and a place where a person can step in.

FIG. 08 · THE LAYERS AROUND A WHILE-LOOPOUTERMOST: THE ONE THE RUN DIDN'T HAVE
Honest framing: this is a venture firm drawing a market map. Read "a stack has already formed around each category" as a thesis about where investable companies sit, not a neutral description of the field. The underlying observation survives the framing — the while-loop is trivial and the surrounding layers are the engineering.
FIG. 09 · TWO LOOPS, ONE RULEINFERENCE TIME · TRAINING TIME
Inference time · changes the work the workweights fixed act verify read result try again Training time · changes the model the weightsupdated each round run trajectories score update weights sample traces → training data · preference pairs · rewards
Not every failure is a training problem. Often the higher-leverage fix is outside the weights: a better tool, clearer state, a more precise action space, a stronger verifier. The diagram is ours; the two-loop framing and the traces-become-training-data link are the post's. Both loops are only as good as their verifier: a test suite on the left, the reward on the right, sometimes the same thing.
TakeawayToday, most agent infrastructure and harnesses can help us run loops. The harder problem is finding a loop worth running, and finding the point before diminishing returns for the task at hand.
$0.00
08 · THE BILL HAS TO BECOME VISIBLE

We run loops the way we once ran cloud instances nobody remembered to turn off.

The agent bills by the token, and the token costs the same whether it moves the score or re-minifies the same HTML for the ninth time. In Li's run, two thirds of the spend bought nothing, and neither the loop nor the person running it knew until the trace was read afterward.

FIG. 10 · THE DASHBOARD NOBODY HASCOST PER ITERATION · PROGRESS PER DOLLAR · A LIVE CURVE
The tile values are the run's own arithmetic on the post's figures (the per-iteration figure is $4.24 spread over roughly nineteen turns, illustrative). Stopping well takes infrastructure: something to meter the spend, something to measure progress against it, and something with enough information to cut the loop off.
THE MISSING PIECE

Cost per iteration. Progress per dollar. A curve you can see while it's still running.

Boring, and necessary. Stopping well isn't something you can prompt into existence: it takes something that meters the spend, something that measures progress against it, and something with enough information to cut the loop off.

THE HOSTS' ASIDE

After the fact is the best anyone offers today.

Decant — the ep005 tool from Dosu, where co-host Taylor Dolezal works, so read this with that in mind — draws the context-window chart with every compaction once a session has ended. Nothing we know of draws the curve above while the loop is running. The lab on this page, and the meter at the top of it, are reconstructions from a finished trace, which is exactly the problem.

"They will be the ones whose builders decided, precisely and in advance, what done costs and what done means."Yoko Li, a16z