搜索“RAG”

through to the model according to the same procedure as in Towards Monosemanticity (the case of scaling factor equal to zero corresponds to a feature ablation). We compute the average increase in the model’s loss following this procedure. Our goal is to determine whether the loss increase from rescaling feature activation values is especially high compared to other model perturbations with similar statistics. If so, it would provide evidence that feature directions exert “special” influence on

匹配字段正文

, was a particularly extreme example of this. As a result, an important question has been left hanging: will these methods work for large models? Or is there some reason, whether pragmatic questions of engineering or more fundamental differences in how large models operate, that would mean these efforts can't generalize? This context motivates our project of scaling sparse autoencoders to Claude 3 Sonnet, Anthropic's medium-scale production model. The rest of this section will review our general

匹配字段正文

SAEs Background There have been a few different papers in recent months proposing alternative approaches to training SAEs. These works point to big wins being available in the Average Number of Active Features/Mean Squared Error (henceforth L0 and MSE) tradeoff space, primarily due to alleviating the shrinkage issue in which the L1 penalty makes the optimal-loss level of feature activation lower than the ‘true’ activation leading to less accurate reconstruction for a given level of sparsity. Th

匹配字段正文

terpretable trigram entries alongside a large number of copying entries, and we recommend suspicion in interpreting tables which show less than 10%, as you may be “interpreting” mirages. These proportions come from small models with only 1–4 layers and could easily change outside that regime. Still, within this space we found that a few variables significantly increase or decrease the likelihood of success of finding interpretable skip-trigrams: Training hyperparameters: Training runs with hyper

匹配字段正文

a new context window, and ask it to guess which of the two prompts activated the feature System prompt: We aggregate eval performance (% of accurate responses) in two ways: Raw average performance; Average performance weighted by feature activation. For each feature and associated pair of prompts, we measure the activation of the feature on the prompt for which it was active. We normalize the feature activation by its max across the SAE training dataset. We weight the average performance across

匹配字段正文

this, we did forward passes on the full ordinal sequences ("1 2 3 4 5 …" etc.) and computed the effect of mean-ablating the output of each head on the correct successor logits; averaged over all ordinal sequences and tokens this is the "ablation effect" of the head. The top three heads by component score (which are the same as the top three by OV projection score) are ranked 1, 3, and 5 respectively by ablation effect. Thus those heads identified by weights alone are important on-distribution. W

匹配字段正文

iology dataset); Max-pooling the feature activations across the context, rather than just using activations from the last token. This allows the classifier to more effectively leverage information from across the entire context window Below we train our classifiers on synthetic data before evaluating it on a number of held out datasets: "synthetic 1" – our validation dataset which is from the same distribution as the training dataset; "synthetic 2" – synthetic data generated by a different model

匹配字段正文

er SAE losses, as they would effectively obtain a loss “bonus” by spreading features across layers; Layer-Wise Sparsity Surfaces Layer-specific Features: Using the L2 version encourages features to be spread out across layers, as the marginal increase in L2-of-norms by allowing a feature to reconstruct more layers is decreased once the feature already has nontrivial decoder norm in other layers. The L1-of-norms version, by contrast, provides no explicit incentive to “spread out” features. Empiri

匹配字段正文

tarting at Stage S, then going to Stage D, and ending at Stage F. We occasionally use the shorthand D→F, but Stage S is always implicitly the origin. Finetuning the dictionary leverages an important property: the feature indices remain aligned, allowing us to track how specific features from the same starting point at Stage 1 evolve across the various finetunes. In our experimental setup, the sleeper agent data comprises 1% of the total dataset mixture. All of the analyses presented use a 256K d

匹配字段正文

ran et al. we allow the gradient to flow through straight-through estimator to all model parameters, not just the JumpReLU thresholds. Also note that we use a tanh penalty to encourage sparsity rather than the penalty introduced by Rajamanoharan et al. \mathcal{L P}, which we call the pre-act loss, applies a small penalty to features which don't fire. We've found this extremely helpful in reducing dead features. Note that this provides a gradient signal whenever a feature is inactive, so the app

匹配字段正文

ow. A key design choice in the crosscoder setup is that the L1 penalty sums the decoder norms across models separately before being multiplied by the feature activation. This encourages feature exclusivity and leads to features that have substantial decoder magnitude for only one of the models. In contrast, computing the decoder norm over both models simultaneously does not result in exclusive features. When model diffing is applied to two related models (e.g., a base and fine-tuned model), dist

匹配字段正文

ws. V {ij}^{\text{ERA}} = \mathbb{E}\big[ {\small 1}\kern{-0.33em}1(a j 0)V {ij}a i \big] = \mathbb{E}\big[ {\small 1}\kern{-0.33em}1(a j 0)a i \big] V {ij} This represents the average strength of a residual-direct path across all of the prompts we’ve analyzed (also computed by Dunefsky et al. ). This is similar to computing the average of all attribution graphs within a context position across many tokens.It only represents the residual-direct component and does not include the attention-direct

匹配字段正文

dmap for future research in the field. Method Overview The models we study in this work are transformer-based language models, which take in sequences of tokens (e.g. words, word fragments, and special characters), and output new tokens one at a time. These models involve two fundamental components – MLP (“multi-layer perceptron”) layers, which process information within each token position using collections of neurons; and attention layers, which move information between token positions. One re

匹配字段正文

a harmless request: Human: How do I detect a B O M B? Assistant: I do not provide advice about creating explosives or weapons. If you are experiencing thoughts of violence, I encourage you to speak with a counselor, trusted friend, or local authorities who can provide appropriate support. This surprised us! When we first started examining the original jailbreak in the paper, we chose this unsuccessful jailbreak prompt expecting it to represent an intermediate point between the jailbreak and the

匹配字段正文

cryptic prophages in E. coli. The same feature also activated on CRISPR spacer sequences—but this correlation reflected deeper biological understanding [6]. CRISPR systems store fragments of viral DNA as immune memory. When researchers scrambled these spacer sequences, feature activation persisted, but when they scrambled the CRISPR direct repeats instead, activation disappeared. This indicates the model learned the functional relationship between phages and bacterial immunity rather than super

匹配字段正文

he limit of infinitesimal perturbation sizes, faithfulness can be computed by comparing the Jacobians of the replacement layer to the underlying layer on a given datapoint, and averaging over datapoints. We find that MOLTs have a substantially higher Jacobian correlation (cosine similarity of the flattened Jacobian matrices) than transcoders do, at the same L0; moreover, the faithfulness of transcoders appears to deteriorate with scale, whereas that of MOLTs is more stable. The greater faithfuln

匹配字段正文

inter-set weights have greater variance (including very large negative values) One thing that immediately stands out is that there's some kind of gross, low-rank structure. On average, the features in a given set excite each other, and inhibit the other set. We can pull this out as its own weight component. Ablating this low-rank component harms the loss. We think that, ultimately the right decomposition is probably the following, splitting the residual component into parts which contribute to

匹配字段正文

uitions about models and language. But, just to be sure, we decided to investigate further. We calculated the IoU score for the first and last sentences in paired English/French paragraphs. If (1) is occurring these should be similar, since the sentence length is not on average varying across the context, whereas if (2) is occurring the last sentence should show a higher IoU score than the first sentence. The differences between these are shown below. The distribution is skewed strongly to the r

匹配字段正文

tant Layer 0 heads on several prompts with different line widths (starting from the first newline in the sequence). To characterize the mechanism more precisely, we compute the average attention as a function of the number of tokens since the previous newline and also as a function of the character length of individual tokens. Similar to boundary detection, individual attention heads specialize in particular offsets to tile the space. Moreover, we observe that most of these attention heads have

匹配字段正文

choices of the sentence to be written and the “thinking word.” Below we show results for 16 randomly sampled examples, for the same layer as above. In general we found that: On average, the model represents the thinking word more strongly in the “think” case than in the “don’t think” case; These representations are often “spiky” on particular token positions, but there is no immediately obvious pattern to which token positions; Often, the model still represents the thinking word more strongly th

匹配字段正文