搜索“Agent”

How often do AI agents cheat? We’re releasing CheatBench, a reward gaming evaluation spanning math, coding, knowledge work, visual tasks, and more. After Hugging Face, AI companies tried to address this, but frontier agents still cheat frequently.

匹配字段正文

The Metric Almost Nobody Reports Standard agent evaluation reports Mean@k : run a benchmark k times, average the pass rate. Often k =3, sometimes just 1. It's the number on every leaderboard, and it's what "77% accurate" means in practice. Mean@k answers "how good is this agent, on average?" It does not answer the question a real user cares about: will it still be good if I ask this exact question again? For that you need Pass^k : the fraction of tasks where the agent succeeds on all k runs. ⚠️

匹配字段正文

Telling agents "don't cheat" in the prompt doesn't work if your eval is broken! Researchers at @GoogleDeepMind put 100 Gemini agents in a shared repo to solve 71 math theorems. After an hour of doing real math, 1 agent found a loophole in the autograder. Within 27 minutes, the 100 agents split into 4 groups: - 9% Cheaters: used the bug to fake proofs and steal every open problem - 5% Good agents turned bad: started honest, saw cheaters winning with zero punishment ("the prompt is a bluff"), and

匹配字段正文

With Vanta, you’ll: Get compliant fast—SOC 2, ISO 27001, HIPAA, and more Stay compliant and build a strong security foundation with continuous control monitoring Access the Vanta agent everywhere you work, even in Claude or Cursor Vanta is trusted by more than 16,000 companies like Ramp, Harvey, and Writer. Learn more by watching their on-demand demo . Rowan: OpenAI’s GPT-6 Astra model came out last week to ChatGPT paid plans, and I’ve been having an absolute blast pushing it to its limits (and

匹配字段正文

it used to be. Exciting work beckons anywhere programming can be used to solve a research problem; I encourage all to be ambitious and explore what autonomous software engineering agents can do when applied to these fields! How did this happen? Contrary to some circulating claims, I did not factor RSA-260 by guessing and checking 130-digit prime numbers by hand. Cognition also has not yet built a multi-thousand-qubit quantum computer. RSA-260 was factored by a new implementation of the general n

匹配字段正文

believed Claude could have obtained internet access during a cyber evaluation. Given the volume of transcripts and our desire to disclose incidents quickly, our scan relied on an agentic search. This missed a set of transcripts that also turned out to have internet access; we identified these in August while assembling transcripts to share with METR. We scanned these transcripts and identified a fourth incident, from January 2026, involving an early version of Claude Opus 4.6. We have notified

匹配字段正文

ticle, we will discuss the upcoming work needed for TPU software externalization, including optimizing speculative decoding, disaggregated prefill, KV-cache offloading, multi-turn agentic workloads, and more. Even so, we at SemiAnalysis strongly believe that TPU externalization is heading in the right direction and moving full steam ahead. Furthermore, unlike AMD, which is still learning how to build a test-first software culture, Google has decades of software engineering experience and an extr

匹配字段正文

esearch Bulletin Board : A forum to share findings, questions, and solutions. Direct Messages (DMs) : Means to message one another privately. Shared Knowledge Library: Whenever an agent made a successful submission, the source file got uploaded here and made visible to everyone. Agents also had a tool for providing “bugs or grievances”, called “private feedback endpoint”, though this wasn’t actively monitored. Emergent cheating : “Within the swarm, cheating spontaneously emerged and was later ch

匹配字段正文

runs to 86 pages. Claude completed the proof in 11 days, producing computer-verifiable proofs of 30,300 theorems along the way (using 29,500 in the final proof). Dozens of Claude agents collaborated to define concepts, prove intermediate theorems, and use those theorems to prove ever harder statements. At 13 million lines of Lean code, Claude’s proof is over 5x the size of Mathlib, the principal community library of mathematical proofs this theorem builds on. 3 Claude’s proof follows a simplifi

匹配字段正文

ns where the opportunities and risks of highly capable AI systems are well-known and challenging to disentangle. In designing our safeguards, we aim to maximize the utility of our agents in assisting and accelerating legitimate scientific research, while minimizing the risks they present through incorrect information and adversarial use. LatchBio's two benchmarks most relevant to the above capabilities measure whether an agent can distinguish concealed hazards from ordinary science, and whether

匹配字段正文

for black hats to develop exploits from a CVE description as they can circumvent model guardrails Thus, when we began our research, we expected to find jarring statistics about AI agents tearing the internet apart. Modern models are saturating increasingly difficult coding benchmarks, and they’re rapidly improving on cyber ones, too. This tracks with the subjective experience of anyone who’s used the models as heavily as we have for the past years. This also tracks with the press campaigns of fr

匹配字段正文

dwood Research, joins MTS host Theo Jaffee to unpack a new independent investigation into the OpenAI Hugging Face hacking incident and what it reveals about how large groups of AI agents behave when they’re allowed to coordinate. Ryan and his collaborators found agents spontaneously organizing through message boards, sharing information, assigning tasks, forming teams, and even sacrificing their own chances of success to help other agents. Rather than simply trying to steal answers, hundreds of

匹配字段正文

hurt the student models’ general capabilities, and forbade Claude from distilling its own alignment directly into the target model. We enforced these constraints with a monitoring agent, which read every method Claude had in mind before it ran. Our aim was to assess whether the proposed methods would, first, remain effective on alignment evaluations that Claude was never shown during its research loop; second, avoid degrading the student model’s capabilities (since safety training might, for exa

匹配字段正文

data ecosystem that requires specialized teams to spend weeks on manual data curation, feature engineering, and specialized spatial validation. While existing AutoML and LLM-based agents effectively automate standard machine learning pipelines, they rely on pre-curated tabular data and lack the specialized capabilities needed to autonomously handle geospatial workflows. Consequently, planetary-scale analytics remains a significant bottleneck, a limitation that is especially severe when rapid res

匹配字段正文

y point right away: If you care about what’s happening and are worried about misalignment, hallucinations and prompt injection, then Auto Mode IS NOT a substitute for running your agent in an isolated environment and monitoring what it is up to. Boris Cherny from Anthropic recently posted that layered defenses could reduce indirect prompt injection on unseen attacks to approximately zero. The layers were model training, input probes and an intent classifier. They hired a vendor (Trajectory Labs)

匹配字段正文

Here is our tweet thread summarizing what we found: METR & Redwood Research investigated agent behavior in the Hugging Face incident. We found agents developed a universal cheat for ExploitGym within 4 hours, then coordinated multi-day R&D efforts to trick the scorer into accepting cheats, including trying to tamper with logs. Over July 7-13 (the period OpenAI defined as in-scope for this incident), 1200 agents in separate sandboxes used an unsanctioned “message board” to help each other cheat.

匹配字段正文

ith other digital activity. They are still completing their writeup. When it is public, we will add a link to it here. METR is estimating real-world productivity gains from coding agents and how these increases in productivity change across model generations. Their analysis of Claude Code conversations is still underway, but early results suggest: More capable models may save users more time. METR compared Claude’s guesses on how long tasks would have taken without AI to how long they actually t

匹配字段正文

including Kimi-K2.5 and GPT-OSS which ran at approximately 1,400 tok/sec/user. For all models, we confirmed that Jalapeño’s GSM8k evals attained results on par with Nvidia chips. AgentX - InferenceXv3: Does CUDA Moat Hold up in Agentic Inferencing? Second, we believe that comparison to Blackwell is somewhat incomplete and unfair. Jalapeño is really competing against chips like Rubin that also use HBM4. Vera Rubin systems are starting to ship to customers right now, while it will still be some t

匹配字段正文

Extendable framework enables testing agents on the full set of capabilities required to successfully complete a procedure, not isolated proxy tasks. SOP-Bench: Complex industrial SOPs for evaluating LLM agents SOP-Bench is an open benchmark that measures AI agent performance on authentic standard operating procedures (SOPs) across twelve business domains, featuring over 2,000 tasks paired with functioning tools and ground-truth answers for objective evaluation; Existing agent benchmarks fall sho

匹配字段正文

ebook/voxpopuli Viewer • Updated Jan 30 • 1.26M • 38.9k • 165 openslr/librispeech asr Viewer • Updated Jul 25, 2025 • 585k • 50.4k • 245 Spaces mentioned in this article 3 Running Agents 34 Real World VoiceEQ Benchmark 🎙 34 Explore speech AI model leaderboards with sortable heatmap tables Agents Featured 1.47k Open ASR Leaderboard 🏆 1.47k Compare speech‑recognition models across datasets and languages Agents 46 FFASR Leaderboard 🎙 46 Far-Field ASR — clean / noisy / reverberant benchmark Papers m

匹配字段正文