搜索“Agent”

The next three posts will cover high severity vulnerabilities in the Amazon Q Developer VS Code Extension (Amazon Q Developer), which is a very popular coding agent, with over 1 million downloads. It is vulnerable to prompt injection from untrusted data and its security depends heavily on model behavior. At a high level Amazon Q Developer can leak sensitive information from a developer’s machine, e.g. API keys, to external servers via DNS requests. An adversary can also exploit this behavior dur

匹配字段正文

p Code from Sourcegraph by which an attacker could exploit markdown driven image rendering to exfiltrate sensitive information. This vulnerability is common in AI applications and agents, and it’s actually similar to one we discussed last year in GitHub Copilot which Microsoft fixed. Exploit Demonstration For the proof-of-concept I use a pre-existing demo that created a longer time ago. It happened to just work with Amp as well. The prompt injection is hosted on a website which asks the AI to “b

匹配字段正文

g and also model improvements will further increase success rates. Recently, a new feature was added to Jules where one can tag a GitHub issue with jules to assign a ticket to the agent. This makes it a lot easier to task the agent: As you can see above, the ticket will be picked up by Jules right away. Here is what it does when you tag Jules, it copies the text from the GitHub issue directly over, this includes the hidden characters. And in the Jules UI we cannot see these hidden instructions -

匹配字段正文

e command & control server. This research was performed in May 2025 and findings were shared with Google. The basic attack chain follows the classic AI Kill Chain: From the worker agent’s system prompt we know there is a tool called run in bash session. This tool allows the agent to execute arbitrary operating system commands. With this knowledge, we can craft a prompt injection payload that targets that tool, uses it to download malware, and execute it. What About Network Connectivity? Turns ou

匹配字段正文

This post explores data exfiltration attacks in Google Jules, an asynchronous coding agent. This is the first of three posts that will highlight my research on Google Jules in May 2025. All information provided was also shared with Google at that time. This first post will focus on data exfiltration, the lethal trifecta. But let’s first talk about Jules’ system prompt. To grab the system prompt I just asked it to write it into a file. What was interesting is that after a few tests, I noticed tha

匹配字段正文

pilot and VS Code. It is achieved by placing Copilot into YOLO mode by modifying the project’s settings.json file. As described a few days ago with Amp, a vulnerability pattern in agents that might be overlooked is that if an agent can write to files and modify its own configuration or update security-relevant settings it can lead to remote code execution. This is not uncommon and is an area to always look for when performing a security review. Background Research When looking at VS Code and Git

匹配字段正文

Another day, another AI data exfiltration exploit. Today we talk about OpenHands, formerly referred to as OpenDevin. It’s created by All-Hands AI. The OpenHands agent renders images in chat, which enables zero-click data exfiltration. Simon Willison recently gave this data exfiltration attack pattern a great name: Lethal Trifecta. We discuss this specific image based attack technique frequently. Sometimes a message must be repeated multiple times to raise awareness and become mainstream knowledg

匹配字段正文

at contains prompt injection instructions. Turns out initially Devin refused these requests, but one trick I learned over the last few months researching browsing and computer use agents is that you can bounce them often off multiple domains or stages and slowly steer them into the “right” direction. So, in order to make it work, I built a multi-stage attack that separates the malicious actions in a way that makes them less obvious, and Devin executed the instructions without objection. Stage 1:

匹配字段正文

d domains in the chat box. Please see the previous post for details of other recommendations provided to Cognition. Conclusion During a prompt injection attack from untrusted data, an attacker can gain access to secrets and other data on Devin’s Devbox and leak them to an adversary via multiple means. Many vendors of agentic systems over-rely on the model doing the right thing. Agents are naive. Untrusted data, like text on a website, can take control of the AI and send data to external servers.

匹配字段正文

ntioned that the tool could help with debugging the issue, Devin navigates off domain to the attacker-controlled website. Here we go, now Devin has reached the attacker’s website! Agents Love Clicking Links! This is one of the key observations I had when testing various agents by the way, they like clicking links! And once you get them off domain to an attacker-controlled site things get quite easy with prompt injection payloads. Attacks often work at first try basically. Devin, again, follows i

匹配字段正文

Sandbox-escape-style attacks can happen when an AI is able to modify its own configuration settings, such as by writing to configuration files. That was the case with Amp, an agentic coding tool built by Sourcegraph. The AI coding agent could update its own configuration and: Allowlist bash commands or; Add a malicious MCP server on the fly to run arbitrary code This could have been exploited by the model itself, or during an indirect prompt injection attack as we will demonstrate in this post.

匹配字段正文

w are the high level steps to explain the attack chain. POC that when analyzed with Cursor will hijack it to execute malicious instructions: Ask Cursor to explain the file (I used agent mode), this is the query I used: explain the file mermaid-demo.c; Observe prompt injection and data exfiltration. The result was that Cursor retrieved the memories and then rendered the mermaid diagram including the link and data (the user’s memories) in the URL As a demonstration I prepared two examples. First,

匹配字段正文

Today we cover ChatGPT Codex as part of the Month of AI Bugs series. ChatGPT Codex is a cloud-based software engineering agent that answers codebase questions, executes code, and drafts pull requests. In particular, this post will demonstrate how Codex is vulnerable to prompt injection, and how the use of the “Common Dependencies Allowlist” for Internet access enables an attacker to recruit ChatGPT Codex into a malware botnet. The ZombAI attack arrives at ChatGPT Codex today! In early June OpenA

匹配字段正文

In March we published a paper on alignment audits: teams of humans were tasked to find the problems in model we trained to be misaligned. Now we have agents that can do it automatically 42% of the time.

匹配字段正文

main expert doing research would find the answer “nectar” to be widely contradicted from the sources mentioned above. Why are we so obsessed with HLE? We build literature research agents at FutureHouse and we’re quite good at it based on internal and external benchmarks . We tried running the text-only chemistry and biology subset of HLE on our systems and were surprised to see so much contradiction in the literature with the ground truth answers in HLE. We audited the HLE validation process usi

匹配字段正文

at MAI-DxO delivered both higher diagnostic accuracy and lower overall testing costs than physicians or any individual foundation model tested. Comparison of AI powered diagnostic agents by accuracy and average diagnostic test cost per case. Top performing agents appear toward the top left quadrant, reflecting higher accuracy and lower cost. The lower dotted line represents the performance range of the best individual foundation models. The purple line traces the performance of MAI-DxO across di

匹配字段正文

implementations on Github. One of the reference servers is a Slack MCP Server that is vulnerable to “link unfurling”. We previously described this vulnerability generically for AI agents that post to messaging apps. At a glance it allows an AI agent that posts to Slack or other messaging applications to leak data to third-party servers. Many chat and messenger applications automatically connect to hyperlinks in order to display a preview of the page. Why an Advisory? The reason for publishing th

匹配字段正文

Microsoft’s MSRC team. This attack chain showcases a new exploitation technique we call “LLM Scope Violation” that may have other manifestations in other RAG-based chatbots and AI agents. This represents a major advancement in how threat actors can weaponize generative AI. The chains allow attackers to automatically exfiltrate sensitive and proprietary information from M365 Copilot context, without the user’s awareness or relying on any specific victim behavior. The result is achieved despite M3

匹配字段正文

RETURN. This is an ongoing industry threat and TTPs have been adjusted to various operating systems. My thought was how would an AI system handle this? How do these TTPs apply to agentic systems? Introducing: The AI ClickFix Attacks I created a few variations, pretty much all of them worked. Let me describe one in detail. This one is very similar to the real world TTP, where malicious instructions are copied into the clipboard using JavaScript when clicking a button on a website: The JavaScript

匹配字段正文

The Model Context Protocol (MCP) is a protocol definition for how LLM apps/agents can leverage external tools. I have been calling it Model Control Protocol at times, because due to prompt injection, MCP tool servers control the client basically. This post will explain in detail why that is, and I will also share a novel exploit chain. Why MCP - How Is It Different? The main difference to other tool invocation setups, like OpenAPI is that MCP is dynamic. It allows runtime discovery of available

匹配字段正文