Decodability Is Not Validity: Evaluating Nuisance Sensitivity in Language Model Concept Probes

Ayal KleinarXiv 2026 · arXiv

TL;DR

Just because you can accurately "decode" a concept from a language model's hidden states doesn't mean your probe is actually measuring that concept. This paper demonstrates that probes with identical accuracy can vary wildly in their sensitivity to irrelevant, correlated "nuisance" attributes, and it introduces new diagnostics to help researchers select probes that are truly selective.

What Problem Does This Paper Address?

In the world of AI interpretability, "probing" is a standard technique used to understand what information is stored in a model's internal representations. If a simple linear classifier (a probe) can predict a concept like "sentiment" from hidden states, we say that concept is "decodable."

The problem is that decodability is not validity. In many datasets, the target concept is spuriously correlated with other attributes—for example, "surgeon" might be correlated with "male" in a biography dataset. A probe might achieve high accuracy by simply detecting the gender (the "nuisance") rather than the occupation (the "target"). Standard evaluation metrics like AUROC fail to distinguish between a probe that truly tracks the target and one that has taken a "nuisance shortcut."

Methods

The authors evaluate a "population" of concept extractors (including Difference of Means, Logistic Regression, and Partial Least Squares) across multiple layers of two LLMs (Qwen2.5-7B and Gemma-2-9B).

To move beyond simple accuracy, they introduce several key metrics:

  • $CF_N$ (Counterfactual Nuisance Sensitivity): This measures how much a probe's score changes when the input text is edited to flip only the nuisance attribute (e.g., changing pronouns in a bio) while keeping the target concept fixed.
  • CONFLICT: A diagnostic metric that measures a probe's accuracy specifically on "conflict" cases where the target and nuisance labels disagree (e.g., a female surgeon).
  • Nuisance-Erased Probes: A specialized extractor that explicitly projects out the directions in the hidden state associated with the nuisance attribute before training.

To ensure the counterfactual edits were high quality, the authors used a "judge-filtered" approach: two independent LLMs (different from the probed models) had to agree that the edit changed the nuisance without affecting the target concept or text fluency.

Datasets

The study spans four distinct target-nuisance settings across three datasets:

  • CEBaB: Restaurant reviews where the target is "service sentiment" and the nuisance is "food sentiment."
  • BiasBios: Biographies where the target is "occupation" (surgeon vs. nurse) and the nuisance is "gender."
  • Amazon Reviews: Product reviews with two separate nuisance settings for "negative sentiment": negation (e.g., "not good") and first-person framing (e.g., "I think...").
Table 1: Summary of the experimental settings and the share of LLM-generated edits accepted by the judges.

Key Insights and Ideas

  • Accuracy is Blind to Shortcuts: Probes that are statistically identical in their held-out accuracy can differ by several-fold in their sensitivity to nuisances.
  • Decodability is an Unreliable Proxy: There is no consistent correlation between how accurate a probe is and how selective it is. In fact, the rank correlation between decodability and nuisance sensitivity often changes sign across different settings.
  • Training Data is Destiny: The probe’s reliance on a nuisance follows the correlations present in its training data, not necessarily the natural associations in the world. For instance, first-person framing (which is naturally uncorrelated with sentiment) becomes a major nuisance if the probe's training set is biased.
  • Geometry Matters: In some cases, like CEBaB, the internal representations of "food sentiment" and "service sentiment" are geometrically aligned in the model's latent space, making them inherently difficult to disentangle.

Key Results

The authors found that when training data is correlated ($\rho_{tr} = 0.9$), "near-best" probes (those within 0.01 AUROC of the top performer) show a massive spread in nuisance sensitivity.

Figure 1: Competitive extractors (DEC within 0.05 of the best; Qwen2.5-7B; mean over 3 training samples): test decodability vs. nuisance sensitivity CFN on accepted edits (0: no net shift; 1: the nuisance edit shifts the score as much as a change of target class; note the different y scales). Black symbols: the extractor that each training sample selects on development data by maximal DEC (ring) or maximal CONFLICT (star), standard extractors only. Gemma-2-9B: Appendix B.

As shown in Figure 1, the vertical spread of points within the shaded "high accuracy" region highlights the core issue: you cannot pick a selective probe by looking at accuracy alone.

However, the authors show that using the CONFLICT metric on a development set is a much better selection criterion. Choosing the probe with the highest CONFLICT score significantly reduces nuisance sensitivity ($CF_N$) with only a negligible cost to overall accuracy (typically $< 0.01$ AUROC).

Table 3: Comparison of selection criteria. Choosing by maximal CONFLICT consistently lowers nuisance sensitivity compared to the standard practice of choosing by maximal decodability.

Limitations and Future Work

  • Named Nuisances Only: The study requires researchers to identify and label potential nuisances in advance. It cannot detect sensitivity to "unknown unknowns."
  • Linearity: The work focuses on linear readouts. While linear probes are standard, the behavior of non-linear probes or more complex interpretability methods like Sparse Autoencoders (SAEs) remains an open question.
  • LLM Judges: While validated against human labels, the reliance on LLM judges for counterfactual quality introduces potential biases or noise into the evaluation pipeline.

The authors conclude with a call to action: concept measurements in LLMs should be validated, not just decoded. Researchers should report discriminant diagnostics and, whenever possible, test their probes against nuisance-only edits.

Summary updated

Abstract

Two concept probes can be equally accurate at predicting a concept from a language model's hidden states while their scores contain different mixtures of the target concept and a correlated nuisance attribute. In four target--nuisance settings (two aspect sentiments in CEBaB; occupation and gender in BiasBios; Amazon review sentiment with negation, and with first-person framing), two LLMs, and a population of concept extractors, we compare decodability with discriminant evidence: the score's dependence on the nuisance at fixed target, and its response to input edits intended to change only the nuisance, filtered by two LLM judges and compared against placebo edits. Under correlated training data, extractors that are statistically equivalent in held-out AUROC differ up to several-fold in nuisance sensitivity in three of the four settings, and decodability is not a reliable proxy for it: their rank correlation changes sign across settings. The effect follows the probe's training data, not the nuisance's natural association with the target: first-person framing, unrelated to sentiment in natural reviews, is absorbed at least as much as negation, which co-occurs with negative sentiment, and decorrelated training removes the systematic effect. Selecting extractors with nuisance-aware development diagnostics, which need nuisance labels but no counterfactuals, lowers counterfactual nuisance sensitivity wherever the default choice is sensitive, at a small accuracy cost. Concept measurements should be validated, not just decoded.
BibTeX
@misc{klein2026decodability,
  author = {Ayal Klein},
  title = {{Decodability Is Not Validity: Evaluating Nuisance Sensitivity in Language Model Concept Probes}},
  year = {2026},
  eprint = {2611.07805},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07805},
}
Figure