TL;DR
This paper reveals that standard attribution graphs for Sparse Autoencoders (SAEs) fail because they ignore "feature crossings"—when an ablation causes downstream features to flip between active and inactive states. By introducing the Margin Graph, which accounts for these flips by tracking how close features are to their activation threshold, researchers can reduce prediction error by an order of magnitude.
What Problem Does This Paper Address?
Mechanistic interpretability researchers use Sparse Autoencoders (SAEs) to decompose the "black box" of language models into thousands of interpretable features. To understand how these features interact, they build attribution graphs—maps showing how a change in one feature (like an ablation) affects features in the next layer.
The current standard approach assumes a "fixed support": it only looks at features that were already active before the ablation. However, this paper argues that this assumption is fundamentally flawed. In reality, zeroing out a single feature often pushes other features across their activation threshold (the "cutoff"). These "crossing" features are not just noise; they carry the majority of the information about how the model responds to an intervention.
Methods
The author introduces the Margin Graph. Unlike standard attribution graphs that only linearize active features, the Margin Graph:
- Calculates the Margin: For TopK SAEs (which keep the $k$ largest activations), the margin is the distance between a feature's preactivation and the $k$-th largest value (the cutoff).
- Predicts First-Order Change: It uses a first-order Taylor expansion to estimate how much every downstream feature's preactivation will change due to an upstream ablation.
- Re-ranks and Flips: It adds this predicted change to the original preactivations and re-ranks them. This allows the graph to predict which features will "enter" the active set and which will "leave" it.
The paper also explores Transport Across Seeds, using sparse nonnegative maps to see if the attribution logic learned in one training run (seed) of an SAE carries over to another.
Datasets
The study utilizes several public and custom datasets:
- Pythia Models: Experiments were conducted on
pythia-70mandpythia-160m. - Public SAEs: Five pairs of public TopK SAEs and one ReLU SAE pair from the EleutherAI and
saprmarksreleases. - Cell F Testbed: A custom testbed of 8 SAE runs trained specifically for this paper on
pythia-70m(layers 3 and 4) to allow for controlled cross-seed comparisons. - The Pile: Evaluations were performed using the validation set of the Pile (uncopyrighted version).
Key Insights and Ideas
- Ablations are Categorical Shifts: Zeroing a feature isn't a tiny nudge; it’s a significant intervention. In over 74% of cases, ablating one feature moves at least one downstream feature across the activation cutoff.
- The Signal is in the Crossings: Features that flip from inactive to active (or vice versa) carry 73% to 84% of the "energy" of the model's response. If your graph doesn't account for these, you are missing most of the story.
- The Margin is the Key Metric: A feature's proximity to the activation threshold (its margin) is the best predictor of whether it will be affected by an upstream change.
- Linearity Holds (Mostly): Surprisingly, despite the discrete nature of TopK activation, the first-order change to preactivations is a very accurate predictor of the final code change, provided you re-rank the features.
Key Results
- Accuracy Boost: On public TopK SAEs, the Margin Graph achieved a median relative error of at most 0.033, compared to 0.247 or higher for standard fixed-support graphs.
- Superiority over Baselines: The Margin Graph outperformed more computationally expensive methods like Integrated Gradients (when those methods were restricted to fixed supports).
- Cross-Seed Explanation: Sparse maps between different training seeds can explain 85% of a graph's energy on active features, suggesting that while features themselves might differ across seeds, the underlying attribution logic is largely preserved.
Limitations and Future Work
- Scope of SAE Types: The study focused heavily on TopK SAEs. While it touched on ReLU SAEs, it did not investigate JumpReLU or transcoders, which are becoming increasingly popular.
- Model Scale: The largest model studied was
pythia-160m. It remains to be seen if these "marginal" effects become even more pronounced or behave differently in billion-parameter models. - Computational Costs: While the paper notes the Margin Graph requires "no added product" (computational cost) over standard methods, it doesn't provide detailed runtime or memory benchmarks for large-scale circuit discovery.
- Multi-layer Chains: The paper found that "decoded middle codes" fail to carry attribution accurately across three layers, suggesting that the residual stream itself is the primary carrier of information, which warrants further investigation into "residual chains."
Summary updated