The author claims that the same model with only the harness changed can produce a 24-point gap
Larsen Cundric said that with the model unchanged and only the harness swapped, evaluation results showed a 24-point gap, and said he had previously always thought the harness was the bottleneck. His tweet cited the Harness Leaderboard released by Joël Niklaus, which used 28,849 trials to compare nine coding agent harnesses, with the model, provider and reasoning effort held constant, three attempts per task, using GLM-5.3-Flash, DeepSeek-V4.1-Flash and Flint, run on Terminal-Bench 2.1 and DeepSWE, with inference costing $4,746.
Why read this
SelectedAI score 70
The author's observation that the same model with only the harness changed produced a 24-point gap shows that coding agent evaluation results depend to a large extent on the outer framework.
Topics
View original postI burned a shitload of compute so you don't have to. Releasing the Harness Leaderboard. Which harness should you run your open model in? I ran 28,849 trials to find out. The leaderboard compares nine coding-agent harnesses on open models. In each comparison only the harness changes: the model, provider and reasoning effort stay fixed, and every task gets three attempts. The trials ran GLM-5.3-Flash, DeepSeek-V4.1-Flash and Flint on Terminal-Bench 2.1 and DeepSWE, for $4,746 of inference. Averaged over the three models, mini-swe-agent (@KLieret), a bash-only agent, solves 51.4% of tasks and @openclaw 27.3%. Four others are statistically tied with mini-swe-agent. One of them is Pi (@pidotdev), at 49.4% for $0.29 per solved task, about 45% less than mini-swe-agent's $0.53. The best harness depends on the model: mini-swe-agent on GLM-5.3-Flash, Hermes Agent (@NousResearch) on DeepSeek-V4.1-Flash and Kimi Code (@Kimi_Moonshot) on Flint. @opencode costs $3.35 per solved task, 11.7 times as much as Pi. See Harness Hub's Harness Lens for what the nine harnesses do differently. It documents the prompts, tools and compaction of 34 coding-agent harnesses and links every claim to its source. Leaderboard: https://t.co/AiWiQtBOz5 Harness Lens: https://t.co/I8jbcWa9JH
View original
I burned a shitload of compute so you don't have to. Releasing the Harness Leaderboard. Which harness should you run your open model in? I ran 28,849 trials to find out. The leaderboard compares nine coding-agent harnesses on open models. In each comparison only the harness changes: the model, provider and reasoning effort stay fixed, and every task gets three attempts. The trials ran GLM-5.3-Flash, DeepSeek-V4.1-Flash and Flint on Terminal-Bench 2.1 and DeepSWE, for $4,746 of inference. Averaged over the three models, mini-swe-agent (@KLieret), a bash-only agent, solves 51.4% of tasks and @openclaw 27.3%. Four others are statistically tied with mini-swe-agent. One of them is Pi (@pidotdev), at 49.4% for $0.29 per solved task, about 45% less than mini-swe-agent's $0.53. The best harness depends on the model: mini-swe-agent on GLM-5.3-Flash, Hermes Agent (@NousResearch) on DeepSeek-V4.1-Flash and Kimi Code (@Kimi_Moonshot) on Flint. @opencode costs $3.35 per solved task, 11.7 times as much as Pi. See Harness Hub's Harness Lens for what the nine harnesses do differently. It documents the prompts, tools and compaction of 34 coding-agent harnesses and links every claim to its source. Leaderboard: https://t.co/AiWiQtBOz5 Harness Lens: https://t.co/I8jbcWa9JH
Body
Oh, so the harness actually matters?
Fascinating.
While everyone was shitting on Grok 4.7, I kept saying the harness was the bottleneck.
Same model and swapping only the harness gives a 24 point swing.
Voilà.