Pretty Linear Liars: Unregularized Probes Favor Intervention-Unfaithful Features

Karl Vilhelmsson Emneby, Julian Yocum, Cameron AllenarXiv 2026 · arXiv

TL;DR

Linear probes—a standard tool for "reading the mind" of AI models—can be deceptive: they often identify features that are decodable but completely ignored by the model's internal logic. This paper demonstrates that unregularized probes (OLS) latch onto low-variance noise, and shows that applying ridge regularization is essential to recovering the "intervention-faithful" directions the model actually uses.

What Problem Does This Paper Address?

In the field of mechanistic interpretability, researchers use linear probing to see if a neural network "knows" a specific concept (like a person's birth year). If a simple linear model can predict the concept from the network's internal activations, we say the concept is "decodable."

However, there is a catch: decodability $\neq$ use. A feature might be present in the activations (perhaps as a byproduct of training) without the model actually using that information to make decisions. To prove a model uses a feature, we need intervention faithfulness: if we manually change that feature in the activations, the model’s final output should change accordingly.

The authors address a specific mystery: why do standard probes often fail causal intervention tests, and how does the choice of fitting procedure (specifically regularization) affect whether a probe finds a "faithful" feature or a "spurious" one?

Methods

The authors compare three ways of finding a "feature direction" in activation space:

  1. Ordinary Least Squares (OLS): Standard linear regression with no regularization.
  2. Ridge Regression: Linear regression with an $L_2$ penalty ($\lambda$) to shrink weights.
  3. Distributed Alignment Search (DAS): A specialized method that finds directions by directly optimizing for intervention effect (the "causal ground truth").

They provide a theoretical proof using the Principal Component (PC) basis of activations. They show that OLS weights are inversely proportional to the variance of a direction. This means OLS is mathematically biased toward magnifying "quiet," low-variance directions that have tiny, spurious correlations with the target—essentially finding "noise" that happens to look like the signal.

Datasets

The study uses two distinct testbeds:

  • Mess3 HMM: A synthetic "toy" transformer trained on a 3-state Hidden Markov Model. This is a "perfect" laboratory because the ground-truth internal states (belief states) are mathematically exact.
  • Llama-2-7B (Birth Year): A real-world large language model probed for its knowledge of birth years using a dataset derived from Wikidata.

Key Insights and Ideas

  • The OLS "Blow-up": In high-dimensional activation spaces, there are thousands of directions that have no causal link to a feature but might have a tiny bit of sampling noise correlated with it. OLS amplifies these directions because they have low variance, leading the probe to point toward "causal dead weight."
  • Regularization as a Causal Filter: Ridge regression acts as a filter. By penalizing large weights, it suppresses the amplification of low-variance noise. This forces the probe to focus on higher-variance directions, which are much more likely to contain the actual features the model uses.
  • Spontaneous Alignment: The most striking finding is that as you increase the regularization strength ($\lambda$), the probe direction rotates away from the OLS noise and begins to align closely with the DAS direction—even though the ridge probe only knows about decodability and DAS only knows about causality.

Key Results

  • OLS is Causal Noise: On Llama-2-7B, the OLS probe direction was nearly orthogonal (84.7°) to the causal DAS direction. Intervening on the OLS direction had virtually zero effect on the model's output, despite having the best "decodability" on the training set.
  • Ridge Recovers Faithfulness: By sweeping for the optimal $\lambda$, the authors found a ridge probe that was much closer to the DAS direction (26.3°) and successfully steered the model's output (halving the intervention loss compared to OLS).
  • Variance Gap: The directions the model actually uses (found by DAS and Ridge) carry orders of magnitude more variance than the directions picked by OLS.

Limitations and Future Work

  • Regression vs. Classification: The theory currently covers regression probes (continuous targets). Most interpretability work uses logistic regression (classification), which requires a similar theoretical extension.
  • Partial Causal Control: Even the best ridge probes don't achieve perfect intervention faithfulness; the intervention loss remains above zero, suggesting that linear probes might still be missing nuances of how models represent information.
  • Standardizing the Sweep: The authors suggest that instead of picking regularization based on validation accuracy (decodability), researchers should pick it based on a sweep of intervention faithfulness.

For those interested in the math, the paper provides a concise derivation in Section 3 showing how the OLS weight $w$ for a direction $v$ is $w = \text{cov}_{vf} / \sigma_v^2$, illustrating the "blow-up" as variance $\sigma_v^2$ approaches zero.

Summary updated

Abstract

Linear probing is a common technique in mechanistic interpretability despite the fact that the decodability of a given feature does not necessarily imply the network uses that feature. Intervention faithfulness, not merely decodability, is the appropriate criterion for whether a feature serves a defined active role in the model. Meanwhile, probe regularization has been used for feature discovery, but the impact of regularization on feature causality remains unclear. We study probe regularization systematically, starting with unregularized ordinary least squares (OLS) regression and considering increasing amounts of regularization via ridge regression. We show theoretically that OLS probes amplify low-variance directions in activation space that have spurious correlations with the target feature, whereas regularization reduces influence from these directions. Empirically, we find in a toy transformer and Llama-2-7B that intervention faithfulness increases up to a point, and then decreases again, and that the optimal regularization strength leads to features bearing striking similarity (in both direction and variance) to those learned by optimizing for intervention outcomes directly. Together, these results provide mechanistic justification for regularizing regression probes.
BibTeX
@misc{emneby2026pretty,
  author = {Karl Vilhelmsson Emneby and Julian Yocum and Cameron Allen},
  title = {{Pretty Linear Liars: Unregularized Probes Favor Intervention-Unfaithful Features}},
  year = {2026},
  eprint = {2611.20371},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20371},
}
Figure