Harness Leaderboard released: the same open model yields significantly different results when the framework is changed
Joël Niklaus released the Harness Leaderboard, using 28,849 trials to compare nine coding agent frameworks on Terminal-Bench 2.1 and DeepSWE, with the model, provider, and reasoning effort held constant, three attempts per task, and a total inference cost of $4,746. The tested models were GLM-5.3-Flash, DeepSeek-V4.1-Flash, and Flint. Averaged across the three models, mini-swe-agent solved 51.4% of tasks, openclaw solved 27.3%, and four other frameworks were statistically on par with mini-swe-agent. The best framework varied by model: mini-swe-agent on GLM-5.3-Flash, Hermes Agent on DeepSeek-V4.1-Flash, and Kimi Code on Flint; Pi cost $0.29 per solved task, while opencode cost $3.35.
Why read this
SelectedAI score 71
After the same open model is switched to different coding agent frameworks, the task resolution rate differs noticeably, and the leaderboard also provides a comparison of inference cost for each task.
Topics
View original postI burned a shitload of compute so you don't have to. Releasing the Harness Leaderboard. Which harness should you run your open model in? I ran 28,849 trials to find out. The leaderboard compares nine coding-agent harnesses on open models. In each comparison only the harness changes: the model, provider and reasoning effort stay fixed, and every task gets three attempts. The trials ran GLM-5.3-Flash, DeepSeek-V4.1-Flash and Flint on Terminal-Bench 2.1 and DeepSWE, for $4,746 of inference. Averaged over the three models, mini-swe-agent (@KLieret), a bash-only agent, solves 51.4% of tasks and @openclaw 27.3%. Four others are statistically tied with mini-swe-agent. One of them is Pi (@pidotdev), at 49.4% for $0.29 per solved task, about 45% less than mini-swe-agent's $0.53. The best harness depends on the model: mini-swe-agent on GLM-5.3-Flash, Hermes Agent (@NousResearch) on DeepSeek-V4.1-Flash and Kimi Code (@Kimi_Moonshot) on Flint. @opencode costs $3.35 per solved task, 11.7 times as much as Pi. See Harness Hub's Harness Lens for what the nine harnesses do differently. It documents the prompts, tools and compaction of 34 coding-agent harnesses and links every claim to its source. Leaderboard: https://t.co/AiWiQtBOz5 Harness Lens: https://t.co/I8jbcWa9JH
View original
I burned a shitload of compute so you don't have to. Releasing the Harness Leaderboard. Which harness should you run your open model in? I ran 28,849 trials to find out. The leaderboard compares nine coding-agent harnesses on open models. In each comparison only the harness changes: the model, provider and reasoning effort stay fixed, and every task gets three attempts. The trials ran GLM-5.3-Flash, DeepSeek-V4.1-Flash and Flint on Terminal-Bench 2.1 and DeepSWE, for $4,746 of inference. Averaged over the three models, mini-swe-agent (@KLieret), a bash-only agent, solves 51.4% of tasks and @openclaw 27.3%. Four others are statistically tied with mini-swe-agent. One of them is Pi (@pidotdev), at 49.4% for $0.29 per solved task, about 45% less than mini-swe-agent's $0.53. The best harness depends on the model: mini-swe-agent on GLM-5.3-Flash, Hermes Agent (@NousResearch) on DeepSeek-V4.1-Flash and Kimi Code (@Kimi_Moonshot) on Flint. @opencode costs $3.35 per solved task, 11.7 times as much as Pi. See Harness Hub's Harness Lens for what the nine harnesses do differently. It documents the prompts, tools and compaction of 34 coding-agent harnesses and links every claim to its source. Leaderboard: https://t.co/AiWiQtBOz5 Harness Lens: https://t.co/I8jbcWa9JH
Body
same open model, different harness => very different results
Joël set out to find which harness actually gets the most out of open models in November 2026