TL;DR
Interpretability claims in machine learning—such as whether a model has "recovered" a specific feature—are often surprisingly fragile, flipping with minor changes to training seeds or thresholds. This paper provides a rigorous mathematical foundation for these claims by introducing regularized, stable versions of common metrics and proving "certificates" that guarantee when a feature recovery claim will survive a change in the model.
What Problem Does This Paper Address?
In the field of mechanistic interpretability, researchers often use tools like Sparse Autoencoders (SAEs) to claim that a model contains specific "features." These claims usually rely on metrics like feature dimensionality ($D_i$) or the Feature Recovery Rate (FRR).
However, these metrics are often discontinuous or highly sensitive. A tiny change in model weights, a different random seed during training, or a slight shift in a cosine threshold can completely reverse a safety claim. The authors note that in previous work, a simple threshold change caused a 24.8% apparent failure in a well-known regularity. This paper asks a fundamental question: When can we actually trust an interpretability measurement?
Methods
The authors take a two-pronged approach: diagnosing the instability of current metrics and proposing stable alternatives.
- Discontinuity Analysis: They prove exactly where the standard feature dimensionality metric ($D_i$) fails. It is discontinuous at the point where a feature vector vanishes while a direction remains orthogonal to all other features.
- Ridge Regularization: To fix this, they introduce a regularized version, $\tilde{D}_i$ (ridge dimensionality), and prove it is Lipschitz stable.
- Sharp Stability Bounds: They derive the "sharp" (best possible) Lipschitz constants for ridge leverage scores. They show that the score changes by at most $\frac{3\sqrt{3}}{8\sqrt{\lambda}}$ times the change in the weights' operator norm.
- Certification: They develop "certificates"—mathematical guarantees—that state: "If feature $i$ is recovered in dictionary $A$, it must also be recovered in dictionary $A'$ if the distance between them is less than $\delta$."
The paper follows a rigorous logical structure, as illustrated in their roadmap:
Notably, the research process itself was novel: the proofs, code, and text were drafted by a pipeline of AI agents (Claude 3.5 models) and verified by an independent "blind" AI reviewer and numerical tests, though the human authors take full responsibility for the content.
Datasets
The study focuses on controlled environments where the "ground truth" is known:
- Toy Models: 12 models trained with varying feature densities ($d=30, n=400$).
- Synthetic Dictionaries: Dictionaries constructed with known features, bisector atoms, and controlled noise levels to test the limits of recovery metrics.
- Dictionary Runs: Small dictionaries trained with three different seeds to measure cross-seed instability.
The authors provide a helpful mapping of how their notation relates to existing literature:
Key Insights and Ideas
- The "Vibes" vs. Math Gap: The paper quantifies just how unstable current methods are. Across three training seeds, 55% of features changed their "recovered" status at standard thresholds, even though the overall area under the recovery curve was stable.
- Universal Stability: The Lipschitz constant for ridge leverage is "shape-free"—it doesn't depend on the number of features or the dimensions of the model, only on the regularization parameter $\lambda$.
- Threshold Sensitivity: The authors show that the common practice of using a hard threshold for feature recovery is the primary source of instability. Moving to smoothed or regularized versions provides the stability needed for reliable safety claims.
Key Results
- Proven Stability: The regularized dimensionality $\tilde{D}_i$ is continuous everywhere, unlike the standard $D_i$.
- Sharp Constants: The paper determines the exact worst-case sensitivity for these metrics. In numerical searches, no pair of models ever exceeded the proved theoretical constants.
- Certificate Performance: The certificates were never wrong in testing. However, they are currently only "informative" for small perturbations. For large jumps (like comparing two completely independent training runs), the atoms move too far (often ~20°) for the current certificates to guarantee recovery non-trivially.
Limitations and Future Work
- Scale: The experiments were limited to toy models and synthetic dictionaries. The authors have not yet tested these certificates on large-scale language models (LLMs).
- The Independent Seed Gap: While the certificates work for small changes (like training checkpoints), they do not yet certify features across independently trained dictionaries where the "feature turnover" is high.
- Open Mathematical Problems: There remains a small gap in the sharp constant for regularized dimensionality on a specific range of parameters ($\rho_1, \rho_2$), which the authors suggest is a candidate for computer-assisted checking.
Summary updated