When can an interpretability claim be trusted? Stability and certification of feature dimensionality, ridge leverage and the feature recovery rate

Ali Tajfar, Behnam TajfararXiv 2026 · arXiv

TL;DR

Interpretability claims in machine learning—such as whether a model has "recovered" a specific feature—are often surprisingly fragile, flipping with minor changes to training seeds or thresholds. This paper provides a rigorous mathematical foundation for these claims by introducing regularized, stable versions of common metrics and proving "certificates" that guarantee when a feature recovery claim will survive a change in the model.

What Problem Does This Paper Address?

In the field of mechanistic interpretability, researchers often use tools like Sparse Autoencoders (SAEs) to claim that a model contains specific "features." These claims usually rely on metrics like feature dimensionality ($D_i$) or the Feature Recovery Rate (FRR).

However, these metrics are often discontinuous or highly sensitive. A tiny change in model weights, a different random seed during training, or a slight shift in a cosine threshold can completely reverse a safety claim. The authors note that in previous work, a simple threshold change caused a 24.8% apparent failure in a well-known regularity. This paper asks a fundamental question: When can we actually trust an interpretability measurement?

Methods

The authors take a two-pronged approach: diagnosing the instability of current metrics and proposing stable alternatives.

  1. Discontinuity Analysis: They prove exactly where the standard feature dimensionality metric ($D_i$) fails. It is discontinuous at the point where a feature vector vanishes while a direction remains orthogonal to all other features.
  2. Ridge Regularization: To fix this, they introduce a regularized version, $\tilde{D}_i$ (ridge dimensionality), and prove it is Lipschitz stable.
  3. Sharp Stability Bounds: They derive the "sharp" (best possible) Lipschitz constants for ridge leverage scores. They show that the score changes by at most $\frac{3\sqrt{3}}{8\sqrt{\lambda}}$ times the change in the weights' operator norm.
  4. Certification: They develop "certificates"—mathematical guarantees—that state: "If feature $i$ is recovered in dictionary $A$, it must also be recovered in dictionary $A'$ if the distance between them is less than $\delta$."

The paper follows a rigorous logical structure, as illustrated in their roadmap:

Roadmap of the paper's proofs and results

Notably, the research process itself was novel: the proofs, code, and text were drafted by a pipeline of AI agents (Claude 3.5 models) and verified by an independent "blind" AI reviewer and numerical tests, though the human authors take full responsibility for the content.

Datasets

The study focuses on controlled environments where the "ground truth" is known:

  • Toy Models: 12 models trained with varying feature densities ($d=30, n=400$).
  • Synthetic Dictionaries: Dictionaries constructed with known features, bisector atoms, and controlled noise levels to test the limits of recovery metrics.
  • Dictionary Runs: Small dictionaries trained with three different seeds to measure cross-seed instability.

The authors provide a helpful mapping of how their notation relates to existing literature:

Notation mapping of the main sources to the paper's terms

Key Insights and Ideas

  • The "Vibes" vs. Math Gap: The paper quantifies just how unstable current methods are. Across three training seeds, 55% of features changed their "recovered" status at standard thresholds, even though the overall area under the recovery curve was stable.
  • Universal Stability: The Lipschitz constant for ridge leverage is "shape-free"—it doesn't depend on the number of features or the dimensions of the model, only on the regularization parameter $\lambda$.
  • Threshold Sensitivity: The authors show that the common practice of using a hard threshold for feature recovery is the primary source of instability. Moving to smoothed or regularized versions provides the stability needed for reliable safety claims.

Key Results

  • Proven Stability: The regularized dimensionality $\tilde{D}_i$ is continuous everywhere, unlike the standard $D_i$.
  • Sharp Constants: The paper determines the exact worst-case sensitivity for these metrics. In numerical searches, no pair of models ever exceeded the proved theoretical constants.
  • Certificate Performance: The certificates were never wrong in testing. However, they are currently only "informative" for small perturbations. For large jumps (like comparing two completely independent training runs), the atoms move too far (often ~20°) for the current certificates to guarantee recovery non-trivially.

Limitations and Future Work

  • Scale: The experiments were limited to toy models and synthetic dictionaries. The authors have not yet tested these certificates on large-scale language models (LLMs).
  • The Independent Seed Gap: While the certificates work for small changes (like training checkpoints), they do not yet certify features across independently trained dictionaries where the "feature turnover" is high.
  • Open Mathematical Problems: There remains a small gap in the sharp constant for regularized dimensionality on a specific range of parameters ($\rho_1, \rho_2$), which the authors suggest is a candidate for computer-assisted checking.

Summary updated

Abstract

Interpretability tools increasingly support safety claims, such as that a sparse dictionary has recovered most of a model's features. Such claims rest on measurements with arbitrary thresholds, and small changes to the model weights, a different training seed or a different threshold can reverse them. We study two widely used measurements in toy models and synthetic dictionaries with known ground truth: the feature dimensionality of Elhage et al. and the feature recovery rate (FRR) with a cosine threshold. We characterize exactly where feature dimensionality is discontinuous and replace it with a regularized (ridge) version with sharp stability bounds: the ridge leverage score changes by at most $3\sqrt{3}/(8\sqrtλ)$ times the operator-norm change of the weights, and this constant cannot be improved; for the regularized dimensionality the stability constant grows like $ρ/4$ in the scale $ρ=R/\sqrtλ$, with explicit upper and lower bounds. We also prove certificates stating when a claim "feature $i$ is recovered" must survive a given change of the dictionary. The certificates held in every test and are informative for small perturbations, but certify no feature non-trivially across independently trained dictionaries; at the standard threshold, whether a feature counts as recovered changed with the training seed for 55 percent of features. Experiments use toy models only. The proofs, code and text were drafted by AI agents directed by the authors and checked by an independent blind AI reviewer and by numerical tests; the authors take full responsibility for the content.
BibTeX
@misc{tajfar2026when,
  author = {Ali Tajfar and Behnam Tajfar},
  title = {{When can an interpretability claim be trusted? Stability and certification of feature dimensionality, ridge leverage and the feature recovery rate}},
  year = {2026},
  eprint = {2611.16190},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16190},
}
Figure