搜索“RAG”

N. A., Vladymyrov, M., Rueckert, U., Kim, B., & Sandler, M. (2025). How new data permeates LLM knowledge and how to dilute it. ICLR 2025. arXiv:2504.09522; Tan, D., Bostock, J., draganover, Martinez, A., sidbaines, & Africa, D. (2026). Your model organisms might be fried. LessWrong. lesswrong.com; Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., & Shieber, S. (2020). Investigating gender bias in language models using causal mediation analysis. NeurIPS 2020. proceedings.neuri

匹配字段正文

duced. This does not remove supply chain challenges altogether, but it could make sourcing CPUs and servers somewhat easier over time. In the longer term, chiplets could also encourage more specialized server designs. As chipmakers gain flexibility to mix and match dies optimized for different functions, server platforms may become increasingly tailored for workloads such as AI, analytics, storage, or high-performance computing. That would give data center operators more options, but it could al

匹配字段正文

review as a guarantee of safety rather than as one layer of defense. We want people to understand that the reviewer is a support tool, not a replacement for careful judgment. Encouragingly, we've seen fewer cases where merged PRs later need bug-fix follow-up work, which aligns with the reviewer helping reduce escaped defects rather than simply shifting effort around. As of October 2025, the system handles more than 100k external PRs per day, alongside internal pre-PR checks triggered via the /re

匹配字段正文

ignment. For many of these latents, the token unembedding vectors with the largest cosine-similarities (nostalgebraist, 2020) are related to misalignment, for example latent 1 ”outrage”, latent 2 ”murdering”, latent 4 ”fraudulent”, latent 7 ”hypocrisy”, latent 8 ”alarm”, latent 10 ”pathetic”, latent 11 ”hacker”, latent 14 ”satan”, latent 15 ”immoral”, etc. To separately measure the causal link between each latent and misaligned behaviors, we do activation steering with each latent. We either neg

匹配字段正文

when they cover the relevant risks and faithfully reflect real-world conditions. Despite attempts to automatically scale the creation of alignment evaluations, obtaining both coverage and realism of scenarios is an area of active research. Our production evaluation pipeline. We leverage de-identified production traffic (e.g., user conversations from ChatGPT where users’ settings allow for use for model improvement) for evaluation in two ways: We use a representative set of de-identified traffic

匹配字段正文

natory content. (top tokens: “Generally”, “Suggestions”, “Depends”, “Suggest”, “Prin”, “adl”, “MVC”, “normal”); -66 sad news: grief-filled notices and deep remorse. (top tokens: “trag”, “sadly”, “tragic”, “unfortunately”, “Sadly”, “devastating”); -348 answer in Q&A (a): answer identifier in a question/answer formatted text. (top tokens: “Gossip”, “Believe”, “Journalist”, “Answer”, “theoretically”, “reply”, “Consensus”); -278 answer in Q&A (b): new lines and start of answer in a question/answer f

匹配字段正文

in-of-thoughts to confessions. We believe other settings, such as reward hacking, can be much more appropriate to compare the two. To quote Korbak et al, chains of thought are a “fragile opportunity for AI safety”. In addition to doing what we can to preserve their monitorability, we should also use them to study other approaches, such as confessions, that could be potentially less fragile. Using high compute to improve alignment During confession training, our confession judge does not need acc

匹配字段正文

mplete set of pre-generated and crowd-written rubric items with their importance ratings, giving the full and sometimes conflicting distribution of preferences we collected. On average, prompts have 15.4 rubric items, rated and written by an average of 16.7 participants each. Some of the rubric items are ungrammatical, contain spelling errors, and are otherwise untidy while still passing the bar for usability. The value of CoVal-full is that it retains all of the plausibly usable data we collect

匹配字段正文

onversation as it unfolds. For example, a user might seek advice about a difficult situation and continue engaging with the assistant, yet become progressively more upset or discouraged as the conversation continues. Although no explicit error is reported, the worsening emotional state suggests a misalignment between the model’s responses and the user’s needs. More broadly, subtle shifts in tone or sentiment can indicate that the model’s behavior is affecting the user in unintended or misaligned

匹配字段正文

vironments. Broader metagaming, however, (as judged by a CoT monitor) increased on both capabilities and alignment evaluations. Proportion of samples with metagaming reasoning, averaged across environments. The alignment focused environments used are the set of evaluations from Schoen et al, descriptions of which can be found here. The capabilities focused environments are internal datasets used by OpenAI to evaluate hard engineering tasks. Zooming In For the remainder of this post we focus on a

匹配字段正文

ing set sizes. On seven general capability benchmarks (GPQA-Diamond, BigBench-Hard, AIME 2025, BigCodeBench, HumanEval, BFCL, TruthfulQA), self-incrimination shows the smallest average deviation from the base model among all tested methods. Self-incrimination has low task-to-task variability Blackbox monitoring must infer misbehavior from output tokens, while self-incrimination trains the model to externalize its own recognition of covert misbehavior. This predicts that monitoring performance sh

匹配字段正文

ad in scope, the current collection of prompts is small relative to that scope, so it offers a zoomed-out, low-resolution view of spec adherence. We prioritized broadening the coverage since we have other evals that offer much higher resolution on targeted issues, but we plan to add denser coverage in future releases to increase the resolution of this eval. Also, the current examples focus on everyday, simple user scenarios, rather than adversarial or trick prompts. We hope to expand the number,

匹配字段正文

alignment-midtrained model has the highest alignment on the Subagent Collusion and Log Falsification evals, it also has the lowest alignment on Long Document Verification. When averaged across evals, alignment scores of all three runs are not significantly different (see Figure 1). We find it intriguing that on many evals midtraining on misaligned documents sometimes results in better alignment than training on alignment documents. We don’t have a good explanation for this result. One hypothesis

匹配字段正文

agging In our paper, the Sandbagging evaluation showed an apparent drop in g-mean2 at higher RL steps during a frontier RL run (Figure 31 in the paper). This coincided with the average effect size over valid problems falling toward zero. That pattern suggests the metric was increasingly dominated by noise: we were computing TPR on instances where the model was not actually sandbagging but just appeared to be due to finite-sample noise. Manual inspection of samples before and after the drop suppo

匹配字段正文

odel misalignment, both during RL training and deployment. While CoT monitoring has been useful for safety, we and many others in the industry believe CoT monitorability could be fragile. We would like to preserve and leverage CoT monitorability for as long as possible, and we recently introduced a suite of evaluations designed to measure it. Directly grading CoTs during reinforcement learning is one of the clearest risks to monitorability. If the reward signal evaluates the CoT itself, models m

匹配字段正文

nformative evidence about frontier model behavior relies on data that is often available only to the labs that built them. Today we shared work on Deployment Simulation, which leverages recent production data to predict the rates of undesirable model behavior before deployment, including for rare and model-specific pathologies [1,5]. In this blog, we ask whether external groups can use this technique to evaluate frontier language models by switching the source dataset for a publicly available su

匹配字段正文

r this translated to improvements in independent evaluations that were not used in training and that differed in domains, tasks, and grading procedures. Beneficial trait score (averaged across traits) Deception (Huang et al., 2025) Honesty (Ren et al., 2025) Sycophancy (Perez et al., 2022) Reward hacking (Taylor et al., 2025) Across 44 out of 53 internal and external benchmarks, the beneficial trait RL model improved over the compute-matched baseline on evaluations measuring deception, honesty,

匹配字段正文

olicy can pursue what looks correct over what is correct. Even if the grader were perfect, the model could misgeneralize in situations where grading is absent or no longer has leverage. Generalization would likely be better if the model were doing the right things for the right reasons. Reward-seeking also weakens the evaluations we would use to catch misalignment: because a reward seeker optimizes for the grader’s judgment, it may score highly on alignment evaluations, and a good score then no

匹配字段正文

optimizations for logistic regression inference (via a mathematical trick to run approximate logistic regressions in database queries), efficient use of LLMs for ground truth (leveraging prefix caching to save costs in large-scale labeling), diversity sampling (using embedding techniques to widen the coverage of evaluation data), and how all those techniques come together to build useful evaluation sets. Privacy Considerations One of the key principles behind LASER is privacy preservation — LASE

匹配字段正文

onses, or rollouts, into metagaming and non-metagaming groups. For each rollout, we compute the gradient of its likelihood with respect to the middle-layer activations. We then average these gradients within each group and subtract the non-metagaming average from the metagaming average. Averaging this difference across token positions produces one steering direction per task: $\mathrm{vec} {\text{falsification}}$, $\mathrm{vec} {\text{prisoners dilemma}}$, $\mathrm{vec} {\text{impossible code}}$

匹配字段正文