Are harnesses worth the cost?
Naresh Jain, founder and CEO of the contract-testing company Specmatic, went in expecting the harness to help. "The results flatly contradicted that expectation and surprised me."
An app passed only if it passed all 178 checks
Jain borrows the terms guides and sensors from Birgitta Böckeler, a distinguished engineer at Thoughtworks. Guides are contracts, examples, and a skill file the agent reads first. Sensors are tests and mocks that flag mismatches during the build.
One Java service for product search. The brief spelled out timeouts, retries, and exact error codes.
Two models, 10 attempts each way. Each attempt ran at low reasoning effort in a fresh container.
One gate. A separate evaluator ran 178 checks with fixtures the harness never saw.
Read the guides
The agent maps each required behavior to a contract and a check before coding.
The flagship model didn't need the help
Jain's exploratory runs with Claude Sonnet 4.6, Claude Opus 4.6, and Gemini 3.8 Flash also finished the task unassisted on the first try.
MeasuredAverages over 10 attempts. Estimated cost prices the recorded tokens at API rates.
The budget model needed it, and still cost less
GPT-6 Luna is OpenAI's model for cost-sensitive, high-volume work. In Jain's words, the harness "did not unlock a capability Luna lacked; it turned a 3-in-10 baseline into 10-for-10 reliability."
Past performances
| Entry | Acc. | Tokens | Time | Cost |
|---|---|---|---|---|
| 1Sol alone | 10/10 | 1,076,371 | 6m 51s | $0.406 |
| 2Sol + harness | 10/10 | 11,331,970 | 19m 53s | $2.679 |
| 3Luna alone | 3/10 | 5,860,736 | 15m 56s | $0.086 |
| 4Luna + harness | 10/10 | 22,933,648 | 35m 00s | $0.273 |
Averages per attempt from the article. Lamps show accepted runs.
Sol alone $0.406, Luna alone $0.086, Luna with harness $0.273
The failed apps broke where systems meet
Stewards' report
Seven runs failed, all of them Luna alone, and each passed at least 144 of the 178 checks.
Timed-out product creation
The brief required 202 and a link to poll.
The seven codes cover all seven failed runs. Our arithmetic
Kafka publication
Three of the seven delivered every message and still broke the wire contract.
Creation timed out, and no monitor link came back.
- The client has nothing to poll
Head-on replay, rebuilt from Jain's written failure descriptions.
- Timeout returns 202 and a monitor linkHeld
- Search failure returns 429 and a waitHeld
- Failed publish returns 503Held
- Publish only the first productHeld
- Kafka matches the wire contractHeld
- Bad media type returns 400Held
Early in the build, every requirement is in view.
The dark rows match the reported misses. The fading is ours.
A requirement can sit in the brief and still drop out of the code
Jain calls the miss an attention lapse and names the fix executable attention. A red sensor puts the dropped requirement back in front of the agent.
People wrote the expected results, and the agent had orders to leave them alone. An agent writing its own tests can repeat one misreading in both.
The bill leaves out the people
100 attempts × $0.133 = $13.30. At $100 an hour, 8.0 minutes of a person's time costs the same.
The harness runs on Specmatic, the tool Jain's company makes, and he notes that equivalent tools could do the same job.
8.0 minutes
Not on the tote board
- Writing contracts, examples, and assertionsNot measured
- Reviewing and maintaining the harnessNot measured
- Finding and diagnosing a failed appNot measured
- Repairing the failed appNot measured
Strong models still need a final gate
Jain closes with a revised hypothesis. "What changes with model capability is not whether independent sensors are needed, but when they run."
Sources and limits
- Naresh Jain, Are Harnesses Worth the Cost?, martinfowler.com, October 13, 2026. Checked October 19, 2026.
- The experiment's application brief.
- Birgitta Böckeler, Harness engineering for coding agent users.
- Naresh Jain's GitHub profile, which lists him as Specmatic's founder.
Jain ran 10 attempts per entry on one Java service, from an unusually detailed brief, and added guides and sensors together.
We didn't rerun the experiment. The replay and infield board are illustrations.