Coding agent harnesses cost more without reliably coding better
A controlled study finds coding-agent harnesses often move benchmark results no more than a rerun, while system prompts and tool descriptions can triple token costs.
Coding-agent makers often present a larger harness — the system prompt, tools and context management wrapped around a language model — as a route to better benchmark scores. A new study suggests the headline advantage is usually much less solid than the marketing, while the token bill can be dramatically higher.1
Researchers Liu and Han ran five models through shipped versions of Claude Code, OpenCode and mini-SWE-agent, using 45 difficult SWE-bench Verified tasks alongside a larger pool of 447 tasks.2 They also repeated configurations to separate a genuine harness effect from ordinary run-to-run variation.
The benchmark gap is often noise
On the larger task pool, the heaviest and lightest harnesses sat less than two percentage points apart for both tested models. Across the hardest tasks, changing harnesses moved results by roughly two to five solved tasks, while simply rerunning the same setup could shift the count by up to three.
That uncertainty showed up in the flip rate: 13% of task outcomes changed between reruns, and the same 13% changed when the researchers swapped harnesses. Changing the model produced a bigger 22% shift. The study therefore treats many small harness-to-harness leads as indistinguishable from a fresh draw of benchmark noise.
OpenCode was the important exception. It trailed the other harnesses by as much as nine points on the larger pool. The researchers link about half of that deficit to output limits: when a model ran out of room mid-step, Claude Code prompted it to continue while OpenCode usually ended the attempt. Tighter local time limits and offline search-tool failures were additional uncontrolled factors.
The system prompt sets the bill
The more consistent difference was cost. The researchers estimate that Claude Code resends about 16,600 system-prompt and tool-description tokens per step, compared with roughly 7,000 for OpenCode and fewer than 1,000 for mini-SWE-agent. Because those instructions are repeatedly included as the conversation grows, a single task can cost up to three times more depending on the harness.
The cheapest option is not automatically the best: mini-SWE-agent can lose context on difficult tasks, while a model that needs more reasoning may take extra steps. But the results challenge the idea that a more elaborate wrapper reliably creates a better coding agent. A bigger harness can buy fewer ways to lose a task — and a much larger bill — rather than a dependable pass-rate boost.
Our opinion
This is the kind of agent comparison that deserves more attention: it measures rerun noise instead of treating one leaderboard result as destiny. The practical lesson is not to abandon polished tools, but to demand matched models, identical task conditions, repeated runs and transparent token costs before believing a claimed harness advantage. Liu and Han have also released their analysis code and data for scrutiny.3
The study is still mainly a SWE-bench investigation, and its model-harness grid is too small to explain every interaction. That makes the conclusion appropriately modest: harness design matters in specific failure modes, but the default assumption should be that a small score lead needs replication before it becomes a product claim.
The hero image is a royalty-free coding photograph by Nemuel Sereti on Pexels.4