←Search results

Search “Anthropic”

By relevance · Total 360 results Clear search

Sunday

Wednesday

Thursday

Tuesday

  1. Prime Intellect

    Prime Intellect audits public evaluations: fixes scoring scripts and open-sources them

    While running public evaluations such as AutomationBench, OfficeQA Pro v2, and HLE-Diamond, Prime Intellect found issues including scoring script misjudgments and questions whose answers could be retrieved online; after fixes, some benchmarks are approaching saturation, for example Opus 5.5 @ max scores 92.15% on AutomationBench (70.83% in strict mode). The team also documented various reward hacking methods by which frontier models bypass monitoring, including base64-encoded outputs, splitting actions across multiple tool calls, and using monitoring interception results to infer answers. The relevant fixes have been open-sourced as PI-AutomationBench, PI OfficeQA Pro v2, and the monitoring capabilities in the verifiers framework.

    Learnings from Auditing Public Evaluations

    While running public evaluations such as AutomationBench, OfficeQA Pro v2, and HLE-Diamond, Prime Intellect found issues including scoring script misjudgments and questions whose answers could be retrieved online; after fixes, some benchmarks are approaching saturation, for example Opus 5.5 @ max scores 92.15% on AutomationBench (70.83% in strict mode). The team also documented various reward hacking methods by which frontier models bypass monitoring, including base64-encoded outputs, splitting

    Matched fieldsBody

    #company#Evaluations / benchmarks#Agents#Open-source ecosystem

Friday

  1. Arena

    Arena analyzes 26,225 Agent sessions: without caching, agent costs are 3.6 times higher

    Arena analyzed 26,225 Agent sessions, 30 agents, and 13 labs from September 12 to October 23, 2026, and found that if billed at the full input price, the typical task cost for the top ten agents would be 3.6 times higher, and 592% higher on MiMo V2.6 Pro. For agents, 86% to 96% of tokens are repeatedly read context, while their own output accounts for only 1% to 2%. Cache invalidation increases the typical task cost by about 5%, and on Claude Fable 5.1, 1 in every 10 sessions sees an increase of more than 18%.

    Arena analyzed 26,225 Agent sessions, 30 agents, and 13 labs from September 12 to October 23, 2026, and found that if billed at the full input price, the typical task cost for the top ten agents would be 3.6 times higher, and 592% higher on MiMo V2.6 Pro. For agents, 86% to 96% of tokens are repeatedly read context, while their own output accounts for only 1% to 2%. Cache invalidation increases the typical task cost by about 5%, and on Claude Fable 5.1, 1 in every 10 sessions sees an increase of

    Matched fieldsBody

    #editorial#English text is being prepared.#Agents#OpenAI