TL;DR
Linear probes—a standard tool for "reading the mind" of AI models—can be deceptive: they often identify features that are decodable but completely ignored by the model's internal logic. This paper demonstrates that unregularized probes (OLS) latch onto low-variance noise, and shows that applying ridge regularization is essential to recovering the "intervention-faithful" directions the model actually uses.
What Problem Does This Paper Address?
In the field of mechanistic interpretability, researchers use linear probing to see if a neural network "knows" a specific concept (like a person's birth year). If a simple linear model can predict the concept from the network's internal activations, we say the concept is "decodable."
However, there is a catch: decodability $\neq$ use. A feature might be present in the activations (perhaps as a byproduct of training) without the model actually using that information to make decisions. To prove a model uses a feature, we need intervention faithfulness: if we manually change that feature in the activations, the model’s final output should change accordingly.
The authors address a specific mystery: why do standard probes often fail causal intervention tests, and how does the choice of fitting procedure (specifically regularization) affect whether a probe finds a "faithful" feature or a "spurious" one?
Methods
The authors compare three ways of finding a "feature direction" in activation space:
- Ordinary Least Squares (OLS): Standard linear regression with no regularization.
- Ridge Regression: Linear regression with an $L_2$ penalty ($\lambda$) to shrink weights.
- Distributed Alignment Search (DAS): A specialized method that finds directions by directly optimizing for intervention effect (the "causal ground truth").
They provide a theoretical proof using the Principal Component (PC) basis of activations. They show that OLS weights are inversely proportional to the variance of a direction. This means OLS is mathematically biased toward magnifying "quiet," low-variance directions that have tiny, spurious correlations with the target—essentially finding "noise" that happens to look like the signal.
Datasets
The study uses two distinct testbeds:
- Mess3 HMM: A synthetic "toy" transformer trained on a 3-state Hidden Markov Model. This is a "perfect" laboratory because the ground-truth internal states (belief states) are mathematically exact.
- Llama-2-7B (Birth Year): A real-world large language model probed for its knowledge of birth years using a dataset derived from Wikidata.
Key Insights and Ideas
- The OLS "Blow-up": In high-dimensional activation spaces, there are thousands of directions that have no causal link to a feature but might have a tiny bit of sampling noise correlated with it. OLS amplifies these directions because they have low variance, leading the probe to point toward "causal dead weight."
- Regularization as a Causal Filter: Ridge regression acts as a filter. By penalizing large weights, it suppresses the amplification of low-variance noise. This forces the probe to focus on higher-variance directions, which are much more likely to contain the actual features the model uses.
- Spontaneous Alignment: The most striking finding is that as you increase the regularization strength ($\lambda$), the probe direction rotates away from the OLS noise and begins to align closely with the DAS direction—even though the ridge probe only knows about decodability and DAS only knows about causality.
Key Results
- OLS is Causal Noise: On Llama-2-7B, the OLS probe direction was nearly orthogonal (84.7°) to the causal DAS direction. Intervening on the OLS direction had virtually zero effect on the model's output, despite having the best "decodability" on the training set.
- Ridge Recovers Faithfulness: By sweeping for the optimal $\lambda$, the authors found a ridge probe that was much closer to the DAS direction (26.3°) and successfully steered the model's output (halving the intervention loss compared to OLS).
- Variance Gap: The directions the model actually uses (found by DAS and Ridge) carry orders of magnitude more variance than the directions picked by OLS.
Limitations and Future Work
- Regression vs. Classification: The theory currently covers regression probes (continuous targets). Most interpretability work uses logistic regression (classification), which requires a similar theoretical extension.
- Partial Causal Control: Even the best ridge probes don't achieve perfect intervention faithfulness; the intervention loss remains above zero, suggesting that linear probes might still be missing nuances of how models represent information.
- Standardizing the Sweep: The authors suggest that instead of picking regularization based on validation accuracy (decodability), researchers should pick it based on a sweep of intervention faithfulness.
For those interested in the math, the paper provides a concise derivation in Section 3 showing how the OLS weight $w$ for a direction $v$ is $w = \text{cov}_{vf} / \sigma_v^2$, illustrating the "blow-up" as variance $\sigma_v^2$ approaches zero.
Summary updated