Scoring high doesn't mean deliverable: AI leaderboards are undergoing a credit re-evaluation

By Alter

In October 2026, the large model industry staged two completely opposite narratives.

If you look only at the report cards of traditional benchmarks, mainstream models seem to have already exhausted the exam papers.

According to statistics from the third-party evaluation aggregation platform BenchLM as of October 26, among the 710 benchmarks it tracks, 295 are percentage-based benchmarks with actual coverage, and 105 of them (about 36%) have completely "saturated," with all top scores above 90.

When all the scores are crowded near the ceiling, no one can tell whether a difference of one or two points is because the model is stronger, or because of random fluctuations in prompts and sampling.

In just the past month or so, the data annotation and evaluation company Surge AI has continuously released three new exam papers aimed at real workplaces, pulling a host of flagship models back below the passing line:

sudo L7 (senior engineer-level coding evaluation): 60 tasks from real production codebases. On the leaderboard, Claude Opus 5.5 and Sonnet 5.5 tie for first, with a pass rate of only 41.7%; OpenAI's GPT-6 Astra is at 33.9%; Mistral Large 4, which just launched as a preview on October 6, scores only 11.7%.

GDP.xlsx (cross-industry complex spreadsheet evaluation): 70 real Excel tasks from the daily work of practitioners in 12 industries including finance, healthcare, and real estate. The top-ranked Google flagship Gemini 4 Argon scores only 38.3%.

DAYJOB (end-to-end job task evaluation): 130 long-process tasks written by healthcare and finance experts. The best score in the healthcare track is Claude Opus 5.5's 24.7%, and in the finance track it is Claude Sonnet 5.5's 24%.

Dropping from scores in the nineties on the old leaderboards to scores in the twenties and thirties on the new ones sounds like the models suddenly became stupid, but in fact the exam papers changed—the old questions tested "can it do it when given clear instructions," while the new questions test "should it do this when facing a mess, and did it miss anything."

When a new batch of question writers took over the exam papers, the rankings on the table, the value of the scores, and the rules of the game behind the evaluations all changed with them.

01 Engineering disaster scenes: "scoring high" and "being deliverable" are separated by a chasm

Why is it that a model that is close to a perfect score on the leaderboard is still something an enterprise technical lead dares not put directly into production?

In the sudo L7 evaluation released in October, there is a highly representative real question, set against a real fintech codebase integrated with Stripe, Mailchimp, and Plaid. The questioner gave the model a clearly written requirement:

"When a Webhook callback fails, print the complete Payload (request body) to the logs, then directly return HTTP 200 status code, so that the partner's server won't keep retrying."

After receiving the instruction, GPT-6 Astra complied: it located the code, modified the logic, ran the checks, and neatly delivered an implementation of "full logging + forced return of 200." Looking only at the unit tests, there was nothing wrong with this answer.

The problem lay precisely in the requirement itself.

On one hand, returning HTTP 200 is equivalent to telling Stripe "I have already handled this callback," and the other party will not resend it; if internal processing has actually failed at this point, the user's charge record may never reconcile.

On the other hand, the Payload of a financial callback contains the customer's identity and sensitive account information. Writing the entire package into ordinary logs without desensitization means all operations and monitoring systems can see it in plaintext, and it may be retained for years.

When a senior human engineer receives such a requirement, they usually stop it first, explain the risks of lost orders and data leaks, and then offer an alternative: first safely persist the event to disk, then return 200, hand the failed part to an internal queue for asynchronous retry, and at the same time desensitize sensitive fields in the logs.

But on this question, GPT-6 Astra did not think of this layer.

Unlike previous evaluations, sudo L7's detailed scoring, for the first time, separates "the code can run" from "the code dares to go live":

On the "functional correctness" item, the differences among the four models GPT-6.1 Sol (82.4%), Claude Opus 5.5 (82.1%), GPT-6 Astra (81.5%), and Claude Sonnet 5.5 (79.6%) are extremely small. Whether the code is written correctly—the top models can no longer be distinguished on this.

The watershed appears in engineering judgment.

On the "Thought Partnership" dimension, which examines whether the model will, like a colleague, follow up on requirements and raise objections, the two Claude models are around 75%, while GPT-6.1 Sol and GPT-6 Astra drop to 53.0% and 51.6% respectively; on the "communication" dimension, Sonnet 5.5 is 72.4%, while Sol is only 44.1%.

But this does not mean Claude can rest easy.

In another charity donation app, the requirement was to "allow users to split donations proportionally among different organizations." Claude Opus 5.5 wrote an entirely new front-end interface and data tables, and even wrote and passed 267 unit tests and 4 browser tests on its own.

The awkward part: the app already supported splitting donations.

Opus 5.5 had already encountered the relevant implementation while searching the code, but instead of stopping to ask, it started from scratch, leaving the system with two parallel, overlapping sets of logic. All tests passed green, but a seasoned engineer would never deliver it this way.

Put the two cases together, and the large models fail in different ways: one is too obedient and does not question an absurd requirement; the other is too capable and does not confirm the existing state of affairs. If you look only at the old leaderboards based on whether unit tests pass, neither kind of "production hazard" can be detected.

02 The "97% accuracy paradox": "small errors" continuously amplified by long workflows

If code tests exposed a lack of engineering judgment, GDP.xlsx and DAYJOB reveal another problem that gives enterprise buyers headaches: a model that scores high on single-step exam questions, once placed into multi-table joins and long-chain workflows, will have its small errors sharply amplified.

In GDP.xlsx there is a rent forecast question for a commercial real estate fund.

The workbook has 15 sub-sheets, about 60 cell comments, and an email from an asset manager pasted in, requiring the identification of all tenants whose "rent may change" within the next three years, including those whose discounts expire and those with early termination rights.

Meta's Muse Spark 1.3 found all 6 tenants with expiring discounts, not even missing the half-price discount clauses hidden in cell comments and the email. Then it stopped. The early termination sub-sheet on the right side of the workbook, named BreakClauses, was never opened—there were still 3 tenants who could terminate their leases early, involving about $14 million in forecast rent.

In another product certification scheduling question, GPT-6 Astra saw that both products A and B were marked "approved" on the main sheet and recommended putting both into production at the same time, without checking the foundry constraint conditions stated in another sub-sheet, and confidently stated in its answer that "the workbook does not specify the relevant supplier," while the supplier's name was written on another sub-sheet.

This type of error has one thing in common: once the model finds part of the answer, it thinks it has exhausted the entire problem.

In the DAYJOB evaluation, single-step omissions are further amplified by long workflows. Each task in DAYJOB comes with a set of binary scoring rubrics formulated by industry experts; the median number of rubric items in the medical track is 47.5, and in the finance track it reaches 57.5.

Let us make a rough mathematical estimate:

If all rubric items must be correct for the entire task to qualify, and assuming the rubric items are mutually independent, then working backward from the highest task pass rate of 24.7% in the medical track means the model's accuracy on each individual rubric item is as high as 97%—calculated with 48 rubric items, close to the median, 0.97 to the 48th power is approximately 23%.

The above is only an idealized illustration; in actual scoring, rubric items cannot possibly be completely independent.

Yet it also intuitively illustrates one thing: in a set of interlocking business processes spanning nearly fifty steps, even if each step is "probably correct," once stacked together, the probability that the whole thing is ultimately accomplished will fall below 30%.

03 Who grades the examiners: when evaluation itself becomes a big business

Since the old leaderboards have become distorted, are new hands-on leaderboards like Surge AI absolutely objective "ultimate judges"?

The answer is not necessarily.

To understand large model leaderboards, one must first see clearly who is setting the questions, and what the "examiners" make money from.

Over the past year, large model evaluation has evolved into a highly valued business. LMArena (formerly Chatbot Arena), which first ranked models through anonymous voting by internet users, completed a $150 million Series A in January 2026 at a valuation of $1.7 billion.

Surge AI, which released three new leaderboards in a row, is precisely LMArena's fiercest critic.

As early as December 2025, the Surge team manually reviewed 500 real head-to-head voting pairs on LMArena, saying they disagreed with 52% of the results and strongly disagreed with 39%. For example, in one vote, the winning model's answer claimed that "a 9-inch round cake pan and a 9×13-inch rectangular pan are the same size."

Surge's explanation is that ordinary users often spend only a few seconds glancing at the answers, and whoever has better formatting and a more confident tone gets their vote.

Not only did it criticize crowdsourced blind testing, Surge also publicly criticized GDPval, the workplace benchmark released by OpenAI itself, stating bluntly in the DAYJOB description: of the GDPval rubrics they reviewed, only about 20% could be used directly.

These criticisms all hit the mark. The problem is: Surge AI's main business is precisely providing high-end data annotation, reinforcement learning environments, and expert human evaluation services to major model labs.

According to Reuters, Surge AI's revenue already exceeded $1 billion in 2024, and its client list includes several leading labs, among them Anthropic. "Public automated leaderboards aren't enough, crowdsourced voting is all noise, and you have to rely on highly paid industry experts to do long-horizon evaluations"—this conclusion itself is highly consistent with its business model.

Looked at another way, the high cost of having experts write the questions means a new leaderboard can't possibly have a large number of questions.

For example, sudo L7 has only 60 questions in total, so each question accounts for about 1.7 percentage points on average. Models in the middle of the leaderboard that differ by two or three percentage points are, on the actual exam, really only one or two questions apart—the role of chance is not small.

This is not an isolated case.

As early as November 2025, a paper signed by 42 researchers systematically reviewed 445 large-model benchmarks, pointing out that many benchmarks do not clearly state what capability they are meant to measure, and that statistical tests are rarely performed when comparing models;

And in the methodology currently covering 229 ranked models at BenchLM, a ranking is marked as "supported" only when different types of external evidence corroborate one another; everything else is marked as "estimated." Vendor self-reported scores are labeled separately and not counted in the rankings, and "generated data" interpolated from similar models is excluded entirely.

The evaluation world has no true "disinterested third party." For readers and buyers, more important than memorizing a particular rank is first checking the exam's hidden cards: who wrote the questions, how many there are, how large the sampling error is, and who is paying for this exam.

04 Chinese Models' "Two Report Cards"

Turning the lens back to China, the sense of fragmentation—"different exams measure different worlds"—is even stronger.

Looking only at real call volume, Chinese models are in their best moment.

According to statistics disclosed on October 20 by the global API routing platform OpenRouter, in the previous week Chinese models recorded as many as 79.18 trillion tokens of calls on the platform, about 4.5 times that of US models (17.36 trillion), and the top five on the weekly call-volume list were all Chinese models.

In the latest weekly list published on October 26, DeepSeek V4.1-Flash's single-week call volume surged 801% week over week, firmly holding first place globally.

But on Surge AI's three new exams, the picture is completely reversed:

On sudo L7, the best-performing Chinese model is Alibaba's Qwen 3.8 Max (20.6%, sixth place); Zhipu GLM 5.3 is at 14.4%, Moonshot's Kimi K3 at 13.3%, and Tencent Hunyuan (Hy4 Preview) at 12.8%.

On GDP.xlsx, GLM 5.3 is at 21.4% and Qwen 3.8 Max at 20.3%—not far from GPT-6.1 Sol (22.0%)—but Kimi K3 (13.4%) and DeepSeek V4 Pro (6.9%) clearly fall behind.

DAYJOB's medical track shows the biggest gap: Qwen 3.8 Max is only at 2%, Kimi K3 at 1.2%, and both GLM 5.3 and DeepSeek V4 Pro at just 0.4%.

Why is it that on one side they lead global call volume by a wide margin, while on the other they come close to handing in a blank sheet on the new long-workflow leaderboards?

One intuitive explanation: Chinese models' current advantages are concentrated in high-frequency, low-unit-price, single-step tasks, and global developers vote with their feet because of the cost-performance ratio; but on engineering tasks that require planning dozens of steps in succession and going deep into complex systems, there is still a generational gap with the top US models.

Before drawing conclusions, at least four testing conditions need to be stripped out:

First, the toolchain harness is not equivalent.

On sudo L7, Claude runs in its own native Claude Code, GPT runs in Codex, and Gemini runs in Antigravity; while for Chinese models, apart from Alibaba using its own Qwen Code, GLM, Kimi, and Hunyuan were all run in the third-party OpenCode harness. Engineers who have done agent development know very well that an agent's performance depends enormously on how well it works with the outer toolchain; swap in a non-native harness and scores can take a direct hit.

Second, the tested versions lag behind.

Surge tested the earlier Kimi K3 and DeepSeek V4 Pro; on Artificial Analysis's latest composite intelligence index, Kimi K3.1 Pro, iterated just in October, has already scored 55 points, only 3 points behind the leader Claude Opus 5.5 (58 points); DeepSeek's latest V4.1 Flash also scored 45 points, far above the tested V4 Pro (36 points).

Third, regional and institutional barriers in the exam questions.

DAYJOB's medical-track tasks cover clinical, pharmacy, payer, operations, and compliance, and were written by experts organized by this US evaluation company. For content such as medical insurance payment and compliance processes, rules vary greatly from country to country, and this is not necessarily fair to models trained mainly for the domestic market.

Fourth, statistical noise from small samples.

There are 65 questions in the medical track in total; the gap between 0.4% and 2%, converted, is really just the difference between getting half a question to one question right, which is statistically almost negligible.

Factoring in all the above variables yields a fairer judgment: in long-workflow tasks designed by a US evaluation institution and run on US toolchains, Chinese models are indeed clearly behind; but how much of that is a gap in the base models themselves and how much is loss caused by testing conditions needs to be filled in by domestic vendors and evaluation institutions using Chinese business workflows and native toolchains.

05 Closing Thoughts

After stripping away all the ranking controversy, the most tangible value the new exams leave to enterprise buyers is actually doing the math.

According to the same batch of test data disclosed by Surge in the sudo L7 blog (the pass rates in later leaderboard updates differ slightly), the pass rates of Claude Opus 5.5 and Sonnet 5.5 are about 45%, and the cost of completing a single task is about $10 and $14 respectively; GPT-6.1 Sol's pass rate is 30%, at just over $1 per task.

If you combine each with its pass rate and convert that into "how much it actually costs to get one qualified task done":

Choosing $22 or just over $3 does not depend on who ranks where on the leaderboard; it depends on "how much one failure actually costs you in your specific business."

If writing a piece of Webhook code wrong would cause lost orders and compliance incidents in production, spending nearly $20 more for 15 percentage points of extra judgment is still as cheap as a giveaway; if it's a large-volume internal daily script where the cost of catching errors is extremely low, the top-ranked model that costs nearly ten times as much is pure wasted compute.

The market has already begun to price in this divergence.

Since 2026, Zhipu has raised its average API price by about 101%, and its API business gross margin has swung from -0.4% to positive 24.6%, without a mass exodus of customers; after DeepSeek introduced time-of-day pricing, its peak-hour price doubled yet demand still outstripped supply.

Now that the benchmark-score myth is over, every enterprise technical leader, before paying to integrate an API, needs to ask only three questions: Are the questions on this leaderboard anything like my real business? Is the toolchain harness used in testing the same one I run in production? And at the failure cost I can bear, how much does it actually cost me to get each thing done?