Abstract
To measure how well pretrained representations encode some linguistic property, it is common to use accuracy of a probe, i.e. a classifier trained to predict the property from the representations. Despite widespread adoption of probes, differences in their accuracy fail to adequately reflect differences in representations. For example, they do not substantially favour pretrained representations over randomly initialized ones. Analogously, their accuracy can be similar when probing for genuine linguistic labels and probing for random synthetic tasks. To see reasonable differences in accuracy with respect to these random baselines, previous work had to constrain either the amount of probe training data or its model size. Instead, we propose an alternative to the standard probes, information-theoretic probing with minimum description length (MDL). With MDL probing, training a probe to predict labels is recast as teaching it to effectively transmit the data. Therefore, the measure of interest changes from probe accuracy to the description length of labels given representations. In addition to probe quality, the description length evaluates "the amount of effort" needed to achieve the quality. This amount of effort characterizes either (i) size of a probing model, or (ii) the amount of data needed to achieve the high quality. We consider two methods for estimating MDL which can be easily implemented on top of the standard probing pipelines: variational coding and online coding. We show that these methods agree in results and are more informative and stable than the standard probes.
Introduction: The Selectivity Problem
In Natural Language Processing, researchers frequently use "probes"—simple classifiers trained on top of frozen representations (like BERT or ELMo)—to determine if those representations encode specific linguistic properties, such as Part-of-Speech (PoS) tags. However, standard accuracy metrics can be deceptive. A high-capacity probe can achieve high accuracy even on random "control tasks" (where labels are assigned randomly) simply by memorizing the mapping. This lack of "selectivity" often forces researchers to manually constrain model sizes or data amounts to obtain meaningful results.
From Accuracy to Description Length
This paper addresses the selectivity problem by moving away from raw accuracy toward a metric that naturally penalizes the "effort" a probe exerts to learn a task. By recasting probing as a data transmission problem, we ask: how many bits does it take for a sender (with access to representations and labels) to transmit the labels to a receiver (who only has the representations)?
As shown in the figure above, while standard probes focus solely on final quality (e.g., accuracy), the Minimum Description Length (MDL) metric incorporates both the final quality and the "hardness" or effort required to achieve it. This effort characterizes either the complexity of the probing model or the amount of data needed to learn the pattern.
Methods for Estimating MDL
We propose two practical methods to estimate MDL that can be implemented on top of existing probing pipelines:
- Variational Coding (The "Rulebook" approach): This uses a two-part code. The sender transmits the model weights (the "rules") and then the labels encoded using that model. If the representations have a clear structure, the rules can be simplified (e.g., via pruning), leading to a shorter total codelength.
- Online Coding (The "Practice" approach): Data is transmitted in blocks. Both parties train on the data seen so far to predict the next block. If the representations are high-quality, the model learns the pattern quickly from few examples, reducing the total bits needed over time.
Empirical Evaluation and Insights
We evaluated these methods using ELMo representations across tasks like PoS tagging (Penn Treebank) and the Edge Probing suite. The results demonstrate that MDL provides a much more stable and honest assessment than accuracy.
Key Findings:
- Linguistic vs. Control Tasks: On PoS tagging, a standard probe might show higher accuracy for a random control task (96.3%) than the base task (93.7%). MDL correctly flips this: the base task requires ~163 kbits, while the random task requires ~267 kbits, identifying the random task as "harder" despite the higher accuracy.
- Stability: As seen in Figure 2(d), MDL is remarkably stable across different random seeds, whereas accuracy can fluctuate. It provides a consistent ranking of layers regardless of the specific probe setup.
- Layer Evolution: MDL clearly shows how representations change. While Layer 0 (embeddings) might have high accuracy, it requires significantly more effort to extract information from than the contextualized Layer 1.
- Model Compression: Using variational coding, we found that probes for linguistic tasks could be pruned significantly (e.g., from 1000 neurons down to 33-75) without losing performance, whereas random tasks required the full, dense model to maintain accuracy.
Conclusion
MDL probing offers a more robust alternative to standard accuracy-based probes. By measuring the "effort" required to extract information, it successfully distinguishes between genuine linguistic knowledge and simple task memorization. While online coding is computationally more expensive, the consistency between the two proposed methods suggests that MDL is a reliable metric for evaluating pretrained representations.
Summary updated