Abstract
Sparse Autoencoders (SAEs) are a primary tool in mechanistic interpretability for disentangling polysemantic activations into interpretable features. However, previous studies observed "pathological" sensitivity to SAE reconstruction errors, where nudging activations along the error direction caused unexpectedly large shifts in model output compared to random baselines. In this work, we identify that these observations were largely artifacts of flawed baseline comparisons that implicitly ablated existing features. We introduce "mixture" baselines that provide a more rigorous point of comparison. Our results show that SAE reconstruction errors are well-behaved and that the sparsity of an SAE ($L_0$) is a key determinant of feature influence, with sparser features acting as more potent "levers" on model behavior.
Introduction
In the field of mechanistic interpretability, researchers aim to understand the internal representations of neural networks. Because individual neurons are often polysemantic—performing multiple unrelated tasks—Sparse Autoencoders (SAEs) are employed to decompose activations into a set of sparse, interpretable directions.
To validate the functional relevance of these directions, researchers use sensitive direction analysis: perturbing a model's internal activations along a specific vector and measuring the resulting change in the final prediction (typically via KL divergence). Previous literature reported a "pathological" finding: perturbations along the SAE reconstruction error vector caused significantly higher KL divergence than perturbations along random directions of the same magnitude. This paper investigates whether this sensitivity reflects a fundamental flaw in SAEs or a methodological error in baseline selection.
Methodology: The Mixture Baseline
The authors identify a critical flaw in previous benchmarking. Earlier experiments used "difference" directions, calculated as the vector from the original activation to a new point. Mathematically, this involves subtracting the original activation, which effectively "ablates" the features currently active in the model. This feature ablation is known to cause extreme shifts in model behavior, making the baseline appear artificially sensitive.
To provide a fair comparison, the authors propose "mixture" baselines:
- Cov-random mixture: The difference between two distinct randomly generated activations, adjusted for the model's covariance.
- Real mixture: The difference between two distinct real activations sampled from the dataset.
Crucially, these mixture baselines do not subtract the current activation, avoiding the ablation effect. Experiments were conducted on GPT-2 Small, focusing on the residual stream at Layer 6 (resid_pre) using the Openwebtext dataset.
Key Results: Resolving the Pathological Error
By applying the mixture baselines, the researchers found that the previously observed "pathology" disappears. In Layer 6, the KL divergence of the SAE reconstruction error aligns closely with the cov-random mixture baseline. This suggests that the error vector is not a uniquely sensitive direction, but rather behaves like a typical change in activation space once the ablation artifact is removed.
Sparsity and Feature Influence
A significant finding of this study is the relationship between an SAE’s sparsity ($L_0$) and its influence on model output. The researchers observed that features from sparser SAEs (those with lower $L_0$) have a significantly greater impact on the model’s predictions than features from denser SAEs.
As shown in the extrapolation analysis, as $L_0$ decreases, the "potency" of each individual feature direction increases. This suggests that sparser dictionaries capture more fundamental "levers" of model behavior. Interestingly, while End-to-End (E2E) SAEs are trained to minimize KL divergence directly, they did not show a stronger effect on model output than traditional "local" SAEs at short perturbation lengths. This may be because E2E features are more isotropic, spreading their influence across more directions.
Conclusion and Future Work
This research demonstrates that SAE reconstruction errors are not inherently pathological; rather, previous interpretations were skewed by flawed baselines. By correcting these baselines, we gain a clearer picture of how SAEs interact with model logic. The discovery that sparsity correlates with feature influence provides a new lens for evaluating SAE quality. Future work should investigate why sensitivity varies across different layers and whether the lower initial sensitivity of E2E SAEs indicates a more robust or simply a more diffuse representation of internal features.
Summary updated