Information-Theoretic Probing with Minimum Description Length

Elena Voita, Ivan Titov

Abstract

To measure how well pretrained representations encode some linguistic property, it is common to use accuracy of a probe, i.e. a classifier trained to predict the property from the representations. Despite widespread adoption of probes, differences in their accuracy fail to adequately reflect differences in representations. For example, they do not substantially favour pretrained representations over randomly initialized ones. Analogously, their accuracy can be similar when probing for genuine linguistic labels and probing for random synthetic tasks. To see reasonable differences in accuracy with respect to these random baselines, previous work had to constrain either the amount of probe training data or its model size. Instead, we propose an alternative to the standard probes, information-theoretic probing with minimum description length (MDL). With MDL probing, training a probe to predict labels is recast as teaching it to effectively transmit the data. Therefore, the measure of interest changes from probe accuracy to the description length of labels given representations. In addition to probe quality, the description length evaluates “the amount of effort” needed to achieve the quality. This amount of effort characterizes either (i) size of a probing model, or (ii) the amount of data needed to achieve the high quality. We consider two methods for estimating MDL which can be easily implemented on top of the standard probing pipelines: variational coding and online coding. We show that these methods agree in results and are more informative and stable than the standard probes.

Introduction: The Limits of Accuracy

In NLP, "probing" is a standard technique to evaluate whether representations (like BERT or ELMo embeddings) contain specific linguistic information. Typically, a small classifier (a probe) is trained on frozen representations to predict properties like part-of-speech tags. However, standard probes often fail to distinguish between a model that truly encodes a property and one that simply allows a probe to memorize labels. Powerful probes can achieve high accuracy even on "control tasks" with random labels, making it difficult to assess the quality of the representation itself. Traditionally, researchers have addressed this by manually restricting probe capacity or data, but these constraints are often arbitrary and lead to inconsistent findings.

Methodology: Probing as Data Transmission

The authors propose recasting probing as a data transmission problem. Instead of measuring accuracy, they measure the Minimum Description Length (MDL): the total number of bits required to transmit labels given the representations. This metric naturally incorporates both the final quality and the "effort" required to achieve it.

Conceptual comparison between standard probing (accuracy) and MDL probing (codelength).

The paper evaluates two practical methods for estimating MDL:

  1. Variational Coding (Model Complexity): This measures the cost of transmitting model parameters plus the data cost. Using Bayesian compression, the probe automatically "prunes" itself; easier tasks result in smaller, lower-cost probes.
  2. Online Coding (Data Efficiency): This measures how quickly a model learns by transmitting data in blocks. If representations encode a property well, the model achieves high compression quickly with minimal data.

Key Results and Stability

The empirical results demonstrate that MDL is a significantly more robust metric than accuracy. For instance, while accuracy might be similar for real linguistic tasks and random control tasks, MDL clearly identifies the control tasks as "harder," requiring much larger models or more data.

Empirical results comparing Variational and Online coding across tasks, learning curves, and stability over random seeds.

Key findings include:

  • Linguistic vs. Random: For PoS tagging, the MDL for a random control task was roughly double that of the real task, even when accuracies were nearly identical (~96% vs ~94%).
  • Layer Ranking: MDL clearly distinguishes between layers (e.g., ELMo's first hidden layer vs. its embedding layer) where standard accuracy often shows negligible differences.
  • Stability: MDL remains remarkably stable across different random seeds, whereas probe accuracy can fluctuate significantly.
  • Trained vs. Random Models: MDL reveals a massive gap between pretrained and randomly initialized models, a distinction that standard probes often obscure.

Conclusion

MDL probing provides a more stable metric for evaluating representations by accounting for the effort required to extract information. While methods like online coding can be computationally expensive, the resulting insights are more reliable and require less manual hyperparameter tuning. Future work may apply MDL to analyze how representations evolve during the training of large language models.

Summary updated

Abstract

To measure how well pretrained representations encode some linguistic property, it is common to use accuracy of a probe, i.e. a classifier trained to predict the property from the representations. Despite widespread adoption of probes, differences in their accuracy fail to adequately reflect differences in representations. For example, they do not substantially favour pretrained representations over randomly initialized ones. Analogously, their accuracy can be similar when probing for genuine linguistic labels and probing for random synthetic tasks. To see reasonable differences in accuracy with respect to these random baselines, previous work had to constrain either the amount of probe training data or its model size. Instead, we propose an alternative to the standard probes, information-theoretic probing with minimum description length (MDL). With MDL probing, training a probe to predict labels is recast as teaching it to effectively transmit the data. Therefore, the measure of interest changes from probe accuracy to the description length of labels given representations. In addition to probe quality, the description length evaluates “the amount of effort” needed to achieve the quality. This amount of effort characterizes either (i) size of a probing model, or (ii) the amount of data needed to achieve the high quality. We consider two methods for estimating MDL which can be easily implemented on top of the standard probing pipelines: variational coding and online coding. We show that these methods agree in results and are more informative and stable than the standard probes.
BibTeX
@inproceedings{voita2020information,
  author = {Elena Voita and Ivan Titov},
  title = {{Information-Theoretic Probing with Minimum Description Length}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=tlbN3IpTTS},
}
Figure