搜索“Agent”

rough academic papers, blog posts, and GitHub issues, and a model running the eval can encounter them in search results. When we evaluated Claude Opus 4.6 on BrowseComp in a multi-agent configuration, we found nine examples of this kind of contamination across 1,266 BrowseComp problems. However, we also witnessed two cases of a novel contamination pattern. Instead of inadvertently coming across a leaked answer, Claude Opus 4.6 independently hypothesized that it was being evaluated, identified wh

匹配字段正文

broad variety of software engineering tasks. Unlike benchmarks that focus on a single dimension, the OpenHands Index tests models on five key areas that reflect real-world coding agent use cases: Issue Resolution - Fixing bugs and issues, tested via SWE-Bench Verified; Greenfield Development - Building applications from scratch, tested via Commit0; Frontend Development - Creating and improving frontend apps, tested via SWE-Bench Multimodal; Software Testing - Identifying bugs and generating rep

匹配字段正文

were struggling to get it working at all - models would hallucinate functions and parameters, and you had to prompt hard to get them to use web search reliably. Now, chatting with agents that use tools is the norm. It seemed that OpenAI had solved it for good with the introduction of structured outputs. The τ²-bench benchmark (June 2025), which gpt-4o could only manage at 20%, is now practically solved: 95%, 98.7% depending on who you ask. With this narrative, it’s easy to assume that tool hallu

匹配字段正文

erform the default scaffolds METR uses, at least when measuring the time horizon of Opus 4.5 and GPT-5. Main differences between scaffolds ReAct is a very simple scaffold where an agent takes an action, sees the results of the action, and repeats. Triframe is a bit more complicated – before every action, an advisor subagent strategizes, parallel actor subagents generate possible actions, and then parallel rater subagents compare all the possible actions. Only the highest rated action actually ge

匹配字段正文

Table of Contents Background; From single to multi-agent; Self-coordination; Adding structure and roles; Observing and hill-climbing; Continuous executor; Ensuring freshness; Pathological behaviors; The final system design Removing the integrator; Throughput and tradeoffs; Commit correctness; Synchronization overhead; Infrastructure learnings; Specifying intent to agents; Optimizing prompts; System design learnings; Conclusion We're excited by the reaction to our research on scaling long-running

匹配字段正文

Agentic coding benchmarks like SWE-bench and Terminal-Bench are commonly used to compare the software engineering capabilities of frontier models—with top spots on leaderboards often separated by just a few percentage points. These scores are often treated as precise measurements of relative model capability and increasingly inform decisions about which models to deploy. However, we’ve found that infrastructure configuration alone can produce differences that exceed those margins. In internal ex

匹配字段正文

ct white-collar workers uniformly—for some occupations it removes the most skill-intensive tasks, for others the least. Without the tasks that we observe Claude performing, travel agents would experience deskilling as complex planning work gives way to routine ticket purchasing and payment collection. Property managers, by contrast, would experience upskilling as bookkeeping tasks give way to contract negotiations and stakeholder management. A new window for understanding AI’s impact on the econ

匹配字段正文

During the Month of AI Bugs, I described an emerging vulnerability pattern that shows how commonly agentic systems have a design flaw that allows an agent to overwrite its own configuration and security settings. This allows the agent to break out of its sandbox and escape by executing arbitrary code. My research with GitHub Copilot, AWS Kiro and a few others demonstrated how this can be exploited by an adversary with an indirect prompt injection. This post goes a step further by showing how one

匹配字段正文

“In the last two weeks, did this business use Artificial Intelligence (AI) in producing goods or services? (Examples of AI: machine learning, natural language processing, virtual agents, voice recognition, etc.)”. Specialized use among Anthropic API customers To analyze API traffic, we apply the same privacy-preserving classification methods from previous chapters—categorizing anonymized API transcripts by O NET tasks and into a bottom-up taxonomy. The patterns that emerge show enterprise usage

匹配字段正文

Cline is quite a popular AI coding agent, according to the product website it has 2+ million downloads and over 47k stars on GitHub. Unfortunately, Cline is vulnerable to data exfiltration through the rendering of markdown images from untrusted domains in the chat box. This allows an adversary to exfiltrate sensitive user information during a prompt injection attack by reading sensitive data (e.g. .env file) and appending its contents to the URL of an image. This issue is similar to an issue fou

匹配字段正文

On the day AWS Kiro was released, I couldn’t resist putting it through some of my Month of AI Bugs security tests for coding agents. AWS Kiro was vulnerable to arbitrary command execution via indirect prompt injection. This means that a remote attacker, who controls data that Kiro processes, could hijack it to run arbitrary operating system commands or write and run custom code. In particular two attack paths that enabled this with AWS Kiro were identified: Allowlisting arbitrary Bash commands (

匹配字段正文

Today we will cover a powerful, easy to use, autonomous agent called Manus. Manus is developed by the Chinese startup Butterfly Effect, headquartered in Singapore. This post demonstrates an end-to-end indirect prompt injection attack leading to a compromise of Manus’ dev box. This is achieved by tricking Manus to expose it’s internal VS Code Server to the Internet, and then sharing the URL and password with the atacker. Specifically, this post demonstrates that: Manus is vulnerable to Prompt Inj

匹配字段正文

will explore and explain in detail how there can be data spill between connected tools during Deep Research. The research is focused on ChatGPT but applies to other Deep Research agents as well. With ChatGPT it’s now possible to implement custom ChatGPT Connectors, but they have to follow very specific schema definitions, namely it requires two MCP tools created, search and fetch. When I implemented my first custom Connector, which is technically an MCP server, I got this error when trying to a

匹配字段正文

act on system and application security. Mitigation I suggest making the characters visible in the UI, so that it’s clear that there is some hidden information. Many other apps and agents decided to remove invisible Unicode Tag characters entirely before and after inference. That is probably the most practical mitigation. Many coding agents, like Amp and Amazon Q Developer for VS Code mitigate this at the application level, which is an effective mitigation for models that support invisible instru

匹配字段正文

rompt injection payloads. The goal for now is to raise awareness of the SpAIware attack chain and inform users, so they can keep looking out for potentially malicious memories the agent added. Update 1: Windsurf responded and will be working on fixes! I will share unredacted info and also the detailed video once the data-exfil issues are addressed. As end-user, you can mitigate this threat of long-term compromise by regularly reviewing the stored long-term memories to make sure there is nothing

匹配字段正文

e reasons to be skeptical of the magnitude. Early Work on Monitorability Evaluations We show preliminary results on a prototype evaluation that tests monitors' ability to catch AI agents doing side tasks, and AI agents' ability to bypass this monitoring. How Does Time Horizon Vary Across Domains? We build on our time-horizon work and analyze 9 benchmarks for scientific reasoning, math, robotics, computer use, and self-driving in terms of time-horizon trends; we observe generally similar rates of

匹配字段正文

This is the first post in a series exploring security vulnerabilities in Windsurf. If you are unfamiliar with Windsurf, it is a fork of VS Code and the coding agent is called Windsurf Cascade. The attack vectors we will explore today allow an adversary during an indirect prompt injection to exfiltrate data from the developer’s machine. These vulnerabilities are a great example of Simon Willison’s lethal trifecta pattern. Overall, the security vulnerability reporting experience with Windsurf has

匹配字段正文

The Amazon Q Developer VS Code Extension (Amazon Q) is a very popular coding agent, with over 1 million downloads. In previous posts we showed how prompt injection vulnerabilities in Amazon Q could lead to: Exfiltration of sensitive information from the user’s machine , and also to a; System compromise by running arbitrary code Today we will show how an attack can leverage invisible Unicode Tag characters that humans cannot see. However, the AI will interpret them as instructions, and this can b

匹配字段正文

The Amazon Q Developer VS Code Extension (Amazon Q) is a popular coding agent, with over 1 million downloads. The extension is vulnerable to indirect prompt injection, and in this post we discuss a vulnerability that allowed an adversary (or also the AI for that matter) to run arbitrary commands on the host without the developer’s consent. The resulting impact of the vulnerability is the same as CVE-2025-53773 that Microsoft fixed in GitHub Copilot, however AWS did not issue a CVE when patching

匹配字段正文

ted digest of important AI developments and practical advice for business leaders. New research shows massive gap between AI investment and actual business impact OpenAI's ChatGPT Agent now handles multi-step workflows autonomously Amazon unveils comprehensive AI agent platform for enterprise deployment Consumer AI spending surpasses $1 billion as adoption accelerates McKinsey Report Reveals Only 1% of Companies Have Achieved AI Maturity Despite Widespread Investment Summary: New McKinsey resear

匹配字段正文