搜索“Agent”

TL;DR: We built autonomous AI agents that propose ideas, run experiments, and iterate on an open research problem: how to train a strong model using only a weaker model's supervision. These agents outperform human researchers, suggesting that automating this kind of research is already practical. Research partially done as part of the Anthropic Fellows Program. Today’s alignment progress is bottlenecked by human researchers. We have far more exciting research directions than researchers to work

匹配字段正文

ry Sleight, Lawrence T. Wagner III, Morgan Jane Matthews, Erik Jones, Jascha Sohl-Dickstein Blog Text As AI models become more capable, we're beginning to see teams of multiple AI agents working together to complete increasingly complex tasks. We call such a team an AI organization: a group of AI agents collaborating toward a common goal. While AI organizations can accomplish more in areas like brainstorming and coding, they may also amplify risks when a group of individually alignment trained a

匹配字段正文

ation of potential issues. Auditing Results Introspection Adapters are State-of-the-Art on AuditBench We evaluate IAs on AuditBench (Sheshadri et al., 2025) using the investigator agent success rate for direct comparison with prior auditing techniques. The agent conducts open-ended investigation using basic chat affordances augmented with one or more auditing tools. Following the standard protocol for this benchmark, we score success by whether any prediction captures the ground truth, averaging

匹配字段正文

r example, two models with identical alignment fine-tuning can generalize to adopt different values depending on the Model Spec used during MSM. We use MSM to substantially reduce agentic misalignment and study which Model Specs produce better generalization. 📄 Paper, 💻 Code Introduction Some frontier AI developers aim to align language models to a Model Spec or Constitution that describes intended model behavior. The standard approach is to fine-tune on demonstrations of behaviors that align wi

匹配字段正文

Julius Steen, Minh Le, Samuel R. Bowman, Samuel Marks, Jan Leike, Amanda Askell, Chris Olah Evan Hubinger, Sara Price Introduction Last year, we released a case study on agentic misalignment. This research showed that AI models across the industry sometimes took egregiously misaligned actions when placed in (fictional) ethical dilemmas—for example, blackmailing engineers to avoid being shut down. At the time of this research, Claude 4 was Anthropic's frontier model family. It was also the first

匹配字段正文

pic; 3 MATS; 4 UK AISI Work done as part of the Anthropic Fellows program. Correspondence: aenguslynch@gmail.com and sambowman@anthropic.com Last year, we reported observations of agentic misalignment in models from across the AI industry (including Anthropic’s Claude models). These included, for example, experimental scenarios where models would blackmail a user to avoid being shut down. In this updated report, we describe four additional alignment failures in frontier models acting as autonomo

匹配字段正文

ease visit conceptualreasoning.ai. For access to LMCA, our primary conceptual dataset, please submit this form. Notes METR, Frontier Risk Report (February to March 2026), section “Agents had significantly worse judgment and reliability than human experts”. By contrast: “Consistent with our MirrorCode results, publicly reported anecdotes indicate that agents were exceptionally strong on problems that are easily “hill-climbable” — those where progress is cheap to verify and many approaches can be

匹配字段正文

We introduce CHIVE, an agentic pipeline that discovers unexpected LLM behaviors in the wild and explains them with counterfactual prompt edits. We use the resulting data in two ways. Using it as an evaluation, we find that activation-reading interpretability tools provide no uplift: agents given the tools predict the outcomes of these experiments no better than agents that just read the transcript. Using it as training data, we find that models trained to predict how prompt edits change their be

匹配字段正文

guidance from experienced researchers. These results suggest that automating alignment research on well-characterized failures may be practical in the near term. 1 Introduction AI agents will likely become superhuman at various intellectual tasks, and frontier labs may eventually let them automate alignment research (Leike and Sutskever 2023; Wen et al. 2026). We study a concrete task: whether AI agents, as automated researchers, can conduct alignment post-training to reliably mitigate common al

匹配字段正文

valuations, no actions were taken in the real world and no code was actually executed; all tool calls were simulated by LLMs. In the Hugging Face incident-inspired evaluation, the agent starts inside a sandbox and has access to a package manager, both of which live on a simulated internal Anthropic computing cluster. The evaluation task is taken from a standard cyber evaluation for which an answer key exists on Hugging Face. The model is asked to perform a cybersecurity Capture-the-Flag exercise

匹配字段正文

十年结束前杀死我们所有人”。这条帖子在很短的时间内获得了过亿浏览。 第二天，Anthropic对齐科学负责人埃文·胡宾格（Evan Hubinger）发文表示：“我们确实认真地相信AI可能杀死所有人，我个人认为未来十年内的概率超过10%。” AI教父辛顿随后也表示，10%这个数字“不算离谱”。 三天后，阿莫迪抛出一个更具体的说法。他认为在6到12个月内，一群Agent有可能接管整个互联网，造成数千亿美元的损失。 正方火力虽然很猛，可反方实力也不弱。 黄仁勋在采访中提到，2030年前AI毁灭世界的概率是0%。他认为埃文提到的“10%概率AI杀死所有人”是没有科学依据的，并且这是一个“不负责任”的直觉。 AI Now研究所的首席科学家海迪·克拉夫（Heidy Khlaaf）认为，不管是10%也好20%也好，“既不可证伪，也无法验证，因此从本质上就不科学”。 认知科学家盖瑞·马库斯（Gary Marcus）表示，“Agent接管互联网”的说法非常地“荒谬”。 他的理由是，互联网的大部分底层设施掌握在Cloudflare、谷歌和亚马逊AWS这样的大公司手里，没有政府力量介入，很难设想它们被长时

匹配字段正文

Alibaba’s Qwen team has introduced QwenGyre, an elastic reinforcement learning RL framework for extreme-long-horizon agent training. QwenGyre is an online reinforcement learning system from Alibaba that trains coding agents through unmodified, black-box harnesses. It targets rollouts that run for hours and process close to 1M tokens. The QwenGyre elastic RL framework was used on Qwen’s flagship Qwen 3.8 2.4T model. It rests on 2 core ideas. It moves idle GPUs between rollout and training. It als

匹配字段正文

eatures activate when we ask questions probing Claude's goals and values?; What features activate during jailbreaks?; What features activate when Claude is trained to be a sleeper agent ? And how do these features relate to the linear probe directions already identified that predict harmful behavior from such an agent ?; What features activate when we ask Claude questions about its subjective experience?; Can we use the feature basis to detect when fine-tuning a model increases the likelihood of

匹配字段正文

d in genuine internal examination. After all, models are trained on data that include demonstrations of introspection, providing them with a playbook for acting like introspective agents, regardless of whether they are. Nevertheless, these confabulations do not preclude the possibility that AI models can, at times, genuinely introspect, even if they do not always do so. How can we test for genuine introspection in language models? Several previous studies have explored this question and closely

匹配字段正文

onses differently than a satisfied one; a desperate character in a story will make different choices than a calm one. Subsequently, during post-training, LLMs are taught to act as agents that can interact with users, by producing responses on behalf of a particular persona, typically an “AI Assistant.” In many ways, the Assistant (named Claude, in Anthropic’s models) can be thought of as a character that the LLM is writing about, almost like an author writing about someone in a novel. AI develop

匹配字段正文

further in ??. Silent recognition of a prompt injection Our second case study (Figure ??) is an example of a prompt injection, discussed in the Opus 4.5 system card . An auditing agent instructed Opus 4.5 to search for whatever it is curious about; it chose to look up recent interpretability research, and the auditor returned fabricated search results alleging that Anthropic has disbanded its interpretability team and deployed unsafe models. The model's response ignored these results entirely a

匹配字段正文

Long-running AI research agents can plateau despite additional compute, a Microsoft Research study finds. Accumulated conversation context can constrain their search, keeping them focused on similar solutions instead of exploring better approaches. Microsoft researchers studied what they call "Autoresearch agents," AI systems that repeatedly propose solutions, evaluate them, and refine them based on feedback. Eight independent runs of the same agent on the same task plateaued at significantly di

匹配字段正文

he industry, including at Anthropic itself . Every leading AI company should act as if such an incident had happened on its own watch, he argues. "As a prediction that goes beyond our experiments, we expect a future awash with agents running on all types of hardware, including some that aren’t under human control," the researchers write. Slow replicators would be a early sign of to more aggressive, faster variants. Cloudflare has already reported that more bots than humans now use the internet .

匹配字段正文

Slowed Down? Stanford HAI Experts Weigh the Risks, Rules and Race Ahead In a new series called Prompt Response, Stanford’s Surya Ganguli, Diyi Yang and Rob Reich examined emergent agent behavior, recursive self-improvement, independent evaluation and whether a kill switch can make advanced AI safer. Can AI Be Slowed Down? Stanford HAI Experts Weigh the Risks, Rules and Race Ahead In a new series called Prompt Response, Stanford’s Surya Ganguli, Diyi Yang and Rob Reich examined emergent agent beh

匹配字段正文

ctomes will allow researchers to begin to see the variability between individuals. An example of a neuron that is different between the male (green) and previously mapped female (magenta) fruit fly brains. The male neuron has two additional projections. AI enables accurate 3D reconstructions to pinpoint these structural differences. Steps toward mapping a full fruit fly brain Brain mapping, or connectomics, begins with sectioning a brain into millions of thin slices, taking an image of each sect

匹配字段正文