TL;DR
Just because you can accurately "decode" a concept from a language model's hidden states doesn't mean your probe is actually measuring that concept. This paper demonstrates that probes with identical accuracy can vary wildly in their sensitivity to irrelevant, correlated "nuisance" attributes, and it introduces new diagnostics to help researchers select probes that are truly selective.
What Problem Does This Paper Address?
In the world of AI interpretability, "probing" is a standard technique used to understand what information is stored in a model's internal representations. If a simple linear classifier (a probe) can predict a concept like "sentiment" from hidden states, we say that concept is "decodable."
The problem is that decodability is not validity. In many datasets, the target concept is spuriously correlated with other attributes—for example, "surgeon" might be correlated with "male" in a biography dataset. A probe might achieve high accuracy by simply detecting the gender (the "nuisance") rather than the occupation (the "target"). Standard evaluation metrics like AUROC fail to distinguish between a probe that truly tracks the target and one that has taken a "nuisance shortcut."
Methods
The authors evaluate a "population" of concept extractors (including Difference of Means, Logistic Regression, and Partial Least Squares) across multiple layers of two LLMs (Qwen2.5-7B and Gemma-2-9B).
To move beyond simple accuracy, they introduce several key metrics:
- $CF_N$ (Counterfactual Nuisance Sensitivity): This measures how much a probe's score changes when the input text is edited to flip only the nuisance attribute (e.g., changing pronouns in a bio) while keeping the target concept fixed.
- CONFLICT: A diagnostic metric that measures a probe's accuracy specifically on "conflict" cases where the target and nuisance labels disagree (e.g., a female surgeon).
- Nuisance-Erased Probes: A specialized extractor that explicitly projects out the directions in the hidden state associated with the nuisance attribute before training.
To ensure the counterfactual edits were high quality, the authors used a "judge-filtered" approach: two independent LLMs (different from the probed models) had to agree that the edit changed the nuisance without affecting the target concept or text fluency.
Datasets
The study spans four distinct target-nuisance settings across three datasets:
- CEBaB: Restaurant reviews where the target is "service sentiment" and the nuisance is "food sentiment."
- BiasBios: Biographies where the target is "occupation" (surgeon vs. nurse) and the nuisance is "gender."
- Amazon Reviews: Product reviews with two separate nuisance settings for "negative sentiment": negation (e.g., "not good") and first-person framing (e.g., "I think...").
Key Insights and Ideas
- Accuracy is Blind to Shortcuts: Probes that are statistically identical in their held-out accuracy can differ by several-fold in their sensitivity to nuisances.
- Decodability is an Unreliable Proxy: There is no consistent correlation between how accurate a probe is and how selective it is. In fact, the rank correlation between decodability and nuisance sensitivity often changes sign across different settings.
- Training Data is Destiny: The probe’s reliance on a nuisance follows the correlations present in its training data, not necessarily the natural associations in the world. For instance, first-person framing (which is naturally uncorrelated with sentiment) becomes a major nuisance if the probe's training set is biased.
- Geometry Matters: In some cases, like CEBaB, the internal representations of "food sentiment" and "service sentiment" are geometrically aligned in the model's latent space, making them inherently difficult to disentangle.
Key Results
The authors found that when training data is correlated ($\rho_{tr} = 0.9$), "near-best" probes (those within 0.01 AUROC of the top performer) show a massive spread in nuisance sensitivity.
As shown in Figure 1, the vertical spread of points within the shaded "high accuracy" region highlights the core issue: you cannot pick a selective probe by looking at accuracy alone.
However, the authors show that using the CONFLICT metric on a development set is a much better selection criterion. Choosing the probe with the highest CONFLICT score significantly reduces nuisance sensitivity ($CF_N$) with only a negligible cost to overall accuracy (typically $< 0.01$ AUROC).
Limitations and Future Work
- Named Nuisances Only: The study requires researchers to identify and label potential nuisances in advance. It cannot detect sensitivity to "unknown unknowns."
- Linearity: The work focuses on linear readouts. While linear probes are standard, the behavior of non-linear probes or more complex interpretability methods like Sparse Autoencoders (SAEs) remains an open question.
- LLM Judges: While validated against human labels, the reliance on LLM judges for counterfactual quality introduces potential biases or noise into the evaluation pipeline.
The authors conclude with a call to action: concept measurements in LLMs should be validated, not just decoded. Researchers should report discriminant diagnostics and, whenever possible, test their probes against nuisance-only edits.
Summary updated