Disentangling Representation Knowledge from Probe Learning

123 concepts90 papers1 proposal

To understand what a deep language model "knows," researchers traditionally rely on diagnostic probes—auxiliary classifiers trained on top of frozen internal representations to predict specific linguistic properties, such as part-of-speech tags or syntactic dependency trees. If a simple probe achieves high accuracy, we typically conclude that the model's representations inherently encode that knowledge.

However, this methodology suffers from a fundamental flaw known as supervised probing bias or the probe's dilemma. Because a probe is itself a trainable machine learning model, it possesses its own learning capacity. When a probe achieves high accuracy, it is often impossible to distinguish whether it is merely extracting a property already utilized by the model, or if it is learning to solve the task independently by exploiting high-dimensional statistical regularities.

This confounding effect undermines our ability to faithfully interpret neural network states, leading to a coordinated effort across the field to design tighter, information-theoretic, and parameter-free evaluation metrics.


The Core Ambiguity of Probe Capacity

The tension between probe expressiveness and interpretability was brought to the forefront by Hewitt et al. and Christopher D. Manning in their development of structural probes. They noted that if a probing model is too expressive (for instance, a deep non-linear network), it can synthesize linguistic features that the underlying model has not actually mastered. In the worst case, a sufficiently powerful probe can memorize the target labels entirely, creating a false positive where the researcher concludes the model understands a concept that it does not.

This circularity problem, as Grzegorz Chrupała argues, makes traditional diagnostic probing for hierarchical structures highly ambiguous. When complex decoders are required to extract a property, we cannot verify if the structural knowledge resides in the original model or was generated during the probing process.

This challenge spans multiple domains:

  • Syntax and Semantics: Conneau et al. demonstrated early on that high-capacity classifiers obscure whether sentence encoders are building robust structural representations or relying on natural language priors. Similarly, Alleman et al. and Niu et al. showed that supervised probes struggle to isolate whether a model truly understands hierarchical phrase structures or if the probe is simply exploiting rich, high-dimensional embeddings.
  • Program Semantics: When analyzing language models trained on code, Jin et al. observed that external classifiers risk learning the underlying execution logic themselves rather than proving that the language model is internally simulating the program.
  • World Models: In evaluating whether models induce internal representations of physical or game rules, Søgaard et al. warned against "probing-induced bias," where a classifier identifies spurious correlations on a game board that the underlying model never actually uses to make decisions.

Information-Theoretic Solutions: Measuring the Effort to Learn

To resolve the probe's dilemma, researchers shifted their focus from raw accuracy to information theory. Instead of asking "Can a probe solve this task?", they ask: "How much effort does the probe require to extract this information?"

Minimum Description Length (MDL)

Voita et al. and Ivan Titov introduced an elegant solution using Minimum Description Length (MDL) to evaluate probes (paper). Under the MDL framework, the quality of a representation is determined by the total codelength required to transmit the target labels given the representations.

$$\text{Total Codelength} = \text{Model Compression Cost} + \text{Data Transmission Cost}$$

If a linguistic property is natively and clearly encoded in the representation, the probe can quickly discover the regularities, requiring very little data and a small model description to achieve high performance (a short codelength). If the probe has to memorize the labels or learn the task from scratch because the representation is uninformative, the codelength will be long. This metric naturally penalizes high-capacity probes that "cheat" by memorizing label associations.

Selectivity and Information Gain

To ensure a probe is discovering rather than learning, researchers use control tasks. A control task pairs the real representations with randomized, synthetic labels that preserve the conditional distribution of the original task but strip away the true linguistic meaning.

Zhu et al. provided an information-theoretic unification of probe selection, proving that the two primary methods for identifying reliable probes—selectivity (the accuracy gap between the real task and a random control task) and information gain—are mathematically equivalent. By measuring this gap, researchers can isolate the specific contribution of the representation from the baseline learnability of the target labels.


Eliminating the Probe: Parameter-Free and Native Alternatives

The most definitive way to eliminate supervised probing bias is to bypass auxiliary classifiers entirely. Modern interpretability research achieves this through parameter-free probing, subnetwork identification, and native model interfaces.

Traditional Probing:
[Frozen Representation] ---> [Trainable Classifier (Probe)] ---> [Prediction]
                                    ^ (Can introduce bias/memorization)

Parameter-Free Probing:
[Frozen Representation] ---> [Geometric / Native Metric] --------> [Prediction]
                                    ^ (Zero learned parameters)

Parameter-Free Probing

Instead of training a neural network to find syntax, parameter-free methods look at the mathematical properties of the model's native components.

  • Perturbed Masking: Wu et al. developed a parameter-free technique that extracts syntax trees directly from BERT. By masking one word and measuring how its absence impacts the contextualized representation of another word, they map the model's internal dependency structure without training a single parameter.
  • Attention Score Parsing: Liu et al. introduced an information-theoretic Bayesian framework (IPBP) that extracts fully labeled grammar trees directly from raw internal attention scores, proving that syntactic structures are accessible without external supervision.
  • Masked Word Prediction: Aoyama et al. bypassed probes by evaluating the model's own layer-wise masked word predictions, revealing how syntactic and semantic knowledge naturally evolves across layers using the model's native output head.

Subnetwork Probing

Rather than adding an external model to the frozen representations, subnetwork probing (Sanh et al.) takes a subtractive approach. It searches for a sparse subnetwork already existing within the pre-trained model weights that can perform the target task. Because it does not introduce new learnable parameters, it remains strictly faithful to the original model's capacity and uncovers exactly where specific knowledge is localized.

Probing via Prompting

Li et al. proposed Probing via Prompting (PP), which replaces diagnostic classifiers with natural language prompts. By forcing the model to express its knowledge through its native text-generation interface, this approach ensures high selectivity, preventing an external probe from independently modeling the task and providing a truer reflection of what the model actually knows.

Supervision-Free Clustering and Alignment

Finally, researchers leverage unsupervised methods to map latent spaces without introducing pre-defined human labels. Shi et al. introduced SyntDEC, a deep clustering framework that groups contextualized representations without supervision, proving that models inherently organize text along syntactic lines.

Similarly, Durrani et al. utilized unsupervised concept discovery to track how fine-tuning reshapes a model's latent space, ensuring that the analysis of how grammar is "forgotten" or specialized remains completely unbiased by the learning capacity of a supervised probe.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 88 papers
  2. Version 1 · 2026-10-04 · 161 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_supervised_probing_bias_1b0c5aed,
  title = {supervised probing bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/supervised-probing-bias/rdr_1b0c5aed9f574e55b207591e83060345},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_supervised_probing_bias_1b0c5aed_v2,
  title = {supervised probing bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/supervised-probing-bias/rdr_1b0c5aed9f574e55b207591e83060345/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake