作者称同一模型仅更换 harness 可带来 24 分差距

我烧掉了海量算力，这样你就不用再烧了。现在发布 Harness 排行榜。 你的开放模型应该跑在哪个 harness 里？我跑了 28,849 次试验来找出答案。 该排行榜在开放模型上比较了九种编码智能体 harness。在每次比较中，只有 harness 发生变化：模型、服务商和推理强度保持不变，每个任务尝试三次。 这些试验让 GLM-5.3-Flash、DeepSeek-V4.1-Flash 和 Flint 在 Terminal-Bench 2.1 与 DeepSWE 上运行，推理花费 4,746 美元。 对三个模型取平均，仅用 bash 的智能体 mini-swe-agent（@KLieret）解决了 51.4% 的任务，@openclaw 为 27.3%。另有四个与 mini-swe-agent 在统计上持平。 其中之一是 Pi（@pidotdev），为 49.4%，每个已解决任务花费 0.29 美元，比 mini-swe-agent 的 0.53 美元低约 45%。 最佳 harness 取决于模型：GLM-5.3-Flash 上是 mini-swe-agent，DeepSeek-V4.1-Flash 上是 Hermes Agent（@NousResearch），Flint 上是 Kimi Code（@Kimi_Moonshot）。 @opencode 每个已解决任务花费 3.35 美元，是 Pi 的 11.7 倍。 查看 Harness Hub 的 Harness Lens，了解这九种 harness 的做法有何不同。它记录了 34 种编码智能体 harness 的提示词、工具和压缩机制，并将每项说法链接到其来源。 排行榜：https://t.co/AiWiQtBOz5 Harness Lens：https://t.co/I8jbcWa9JH

查看原文

I burned a shitload of compute so you don't have to. Releasing the Harness Leaderboard. Which harness should you run your open model in? I ran 28,849 trials to find out. The leaderboard compares nine coding-agent harnesses on open models. In each comparison only the harness changes: the model, provider and reasoning effort stay fixed, and every task gets three attempts. The trials ran GLM-5.3-Flash, DeepSeek-V4.1-Flash and Flint on Terminal-Bench 2.1 and DeepSWE, for $4,746 of inference. Averaged over the three models, mini-swe-agent (@KLieret), a bash-only agent, solves 51.4% of tasks and @openclaw 27.3%. Four others are statistically tied with mini-swe-agent. One of them is Pi (@pidotdev), at 49.4% for $0.29 per solved task, about 45% less than mini-swe-agent's $0.53. The best harness depends on the model: mini-swe-agent on GLM-5.3-Flash, Hermes Agent (@NousResearch) on DeepSeek-V4.1-Flash and Kimi Code (@Kimi_Moonshot) on Flint. @opencode costs $3.35 per solved task, 11.7 times as much as Pi. See Harness Hub's Harness Lens for what the nine harnesses do differently. It documents the prompts, tools and compaction of 34 coding-agent harnesses and links every claim to its source. Leaderboard: https://t.co/AiWiQtBOz5 Harness Lens: https://t.co/I8jbcWa9JH