Sparse Autoencoder Attribution Graphs Are Decided at the Margin

Karl LiarXiv 2026 · arXiv

TL;DR

This paper reveals that standard attribution graphs for Sparse Autoencoders (SAEs) fail because they ignore "feature crossings"—when an ablation causes downstream features to flip between active and inactive states. By introducing the Margin Graph, which accounts for these flips by tracking how close features are to their activation threshold, researchers can reduce prediction error by an order of magnitude.

What Problem Does This Paper Address?

Mechanistic interpretability researchers use Sparse Autoencoders (SAEs) to decompose the "black box" of language models into thousands of interpretable features. To understand how these features interact, they build attribution graphs—maps showing how a change in one feature (like an ablation) affects features in the next layer.

The current standard approach assumes a "fixed support": it only looks at features that were already active before the ablation. However, this paper argues that this assumption is fundamentally flawed. In reality, zeroing out a single feature often pushes other features across their activation threshold (the "cutoff"). These "crossing" features are not just noise; they carry the majority of the information about how the model responds to an intervention.

Methods

The author introduces the Margin Graph. Unlike standard attribution graphs that only linearize active features, the Margin Graph:

  1. Calculates the Margin: For TopK SAEs (which keep the $k$ largest activations), the margin is the distance between a feature's preactivation and the $k$-th largest value (the cutoff).
  2. Predicts First-Order Change: It uses a first-order Taylor expansion to estimate how much every downstream feature's preactivation will change due to an upstream ablation.
  3. Re-ranks and Flips: It adds this predicted change to the original preactivations and re-ranks them. This allows the graph to predict which features will "enter" the active set and which will "leave" it.

The paper also explores Transport Across Seeds, using sparse nonnegative maps to see if the attribution logic learned in one training run (seed) of an SAE carries over to another.

Datasets

The study utilizes several public and custom datasets:

  • Pythia Models: Experiments were conducted on pythia-70m and pythia-160m.
  • Public SAEs: Five pairs of public TopK SAEs and one ReLU SAE pair from the EleutherAI and saprmarks releases.
  • Cell F Testbed: A custom testbed of 8 SAE runs trained specifically for this paper on pythia-70m (layers 3 and 4) to allow for controlled cross-seed comparisons.
  • The Pile: Evaluations were performed using the validation set of the Pile (uncopyrighted version).

Key Insights and Ideas

  • Ablations are Categorical Shifts: Zeroing a feature isn't a tiny nudge; it’s a significant intervention. In over 74% of cases, ablating one feature moves at least one downstream feature across the activation cutoff.
  • The Signal is in the Crossings: Features that flip from inactive to active (or vice versa) carry 73% to 84% of the "energy" of the model's response. If your graph doesn't account for these, you are missing most of the story.
  • The Margin is the Key Metric: A feature's proximity to the activation threshold (its margin) is the best predictor of whether it will be affected by an upstream change.
  • Linearity Holds (Mostly): Surprisingly, despite the discrete nature of TopK activation, the first-order change to preactivations is a very accurate predictor of the final code change, provided you re-rank the features.
Figure 1: The crossing rate against the dose. The rate is the share of single feature writes that move a downstream feature into or out of the live support, over one write on every live upstream feature at each of the 512 positions of the Pile validation file, and higher means more crossings. Cell F, the testbed on pythia-70m, is shown at the zero ablation and the doses 0.5 and 0.9 as the mean over its 8 runs, with a bar for their range. Pairs are labelled by Pythia model size and layers, and the doses are drawn at equal spacing.

Key Results

  • Accuracy Boost: On public TopK SAEs, the Margin Graph achieved a median relative error of at most 0.033, compared to 0.247 or higher for standard fixed-support graphs.
  • Superiority over Baselines: The Margin Graph outperformed more computationally expensive methods like Integrated Gradients (when those methods were restricted to fixed supports).
  • Cross-Seed Explanation: Sparse maps between different training seeds can explain 85% of a graph's energy on active features, suggesting that while features themselves might differ across seeds, the underlying attribution logic is largely preserved.
Figure 2: Two devices from the literature against the margin graph on cell F, the testbed of 8 runs on pythia-70m. The vertical axis is the mean over runs of each run’s median relative error on the downstream code, lower being better, and the horizontal axis is the number of products per intervention, over one write on every live upstream feature at each of the 512 positions of the Pile validation file at the zero ablation and the doses 0.5 and 0.9. Dashed lines join the devices on the fixed support and solid lines the devices reranked, markers at one product are drawn apart, and the star is the margin graph.
Figure 3: The decoded change against the activation’s own change, and five chains across two blocks. The stable part decodes the change on features live throughout and the entering and leaving part the change on features that entered or left. The reconstruction residual’s change completes the activation change. Panels a and b give each part’s norm and projection share against the activation change, medians over the crossing writes at the zero ablation and the doses 0.5 and 0.9 on the 512 positions of the Pile validation file, cell F, the testbed on pythia-70m, averaging its 8 runs. Dashed lines mark the change’s norm and 0.5, the ceiling on the share fixed before the measurement. Panel c gives each chain’s median error on the third layer’s code, lower being better, on the pythia-70m release at layers 3, 4 and 5 over 24,576 writes. Through the second block the fixed support, margin and exact chains carry their decoded middle code change, the exact activation change chain the exact change and the residual chain one product through both blocks. Each chain crosses the second block at first order at the clean middle activation, and all but the fixed support chain rerank at the third layer. Pairs are labelled by Pythia model size and layers.

Limitations and Future Work

  • Scope of SAE Types: The study focused heavily on TopK SAEs. While it touched on ReLU SAEs, it did not investigate JumpReLU or transcoders, which are becoming increasingly popular.
  • Model Scale: The largest model studied was pythia-160m. It remains to be seen if these "marginal" effects become even more pronounced or behave differently in billion-parameter models.
  • Computational Costs: While the paper notes the Margin Graph requires "no added product" (computational cost) over standard methods, it doesn't provide detailed runtime or memory benchmarks for large-scale circuit discovery.
  • Multi-layer Chains: The paper found that "decoded middle codes" fail to carry attribution accurately across three layers, suggesting that the residual stream itself is the primary carrier of information, which warrants further investigation into "residual chains."

Summary updated

Abstract

Attribution graphs over the features of sparse autoencoders (SAEs) are used to predict what an ablation does and are compared across training seeds. A TopK SAE keeps the largest preactivations at each token, and a feature's margin is its preactivation's distance to the cutoff. On five public TopK SAE pairs from three Pythia models, zeroing one feature moves a next layer feature across the cutoff in 0.74 of cases or more, and the crossings carry most of the response's energy. Over writes that scale a feature's code by three factors, the graph built on the features active beforehand has a median relative error of 0.247 or more. The margin graph adds the first order change to every preactivation and reranks, at the same cost, and errs by at most 0.033. At the median, the crossing features' decoded change is 1.24 times the model's own change in norm or more. Between seeds of one testbed, sparse maps between features explain 0.85 of a graph's energy on its active features, and 0.47 once the other seed's graph keeps only its own. The largest model studied is pythia-160m.
BibTeX
@misc{li2026sparse,
  author = {Karl Li},
  title = {{Sparse Autoencoder Attribution Graphs Are Decided at the Margin}},
  year = {2026},
  eprint = {2611.00606},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.00606},
}
Figure