Measuring Memorization Effect in Word-Level Neural Networks Probing

Rudolf Rosa, Tomás Musil, David Marecek

Abstract

Multiple studies have probed representations emerging in neural networks trained for end-to-end NLP tasks and examined what word-level linguistic information may be encoded in the representations. In classical probing, a classifier is trained on the representations to extract the target linguistic information. However, there is a threat of the classifier simply memorizing the linguistic labels for individual words, instead of extracting the linguistic abstractions from the representations, thus reporting false positive results. While considerable efforts have been made to minimize the memorization problem, the task of actually measuring the amount of memorization happening in the classifier has been understudied so far. In our work, we propose a simple general method for measuring the memorization effect, based on a symmetric selection of comparable sets of test words seen versus unseen in training. Our method can be used to explicitly quantify the amount of memorization happening in a probing setup, so that an adequate setup can be chosen and the results of the probing can be interpreted with a reliability estimate. We exemplify this by showcasing our method on a case study of probing for part of speech in a trained neural machine translation encoder.

Introduction

In Natural Language Processing (NLP), "probing" has become a standard diagnostic tool to interpret the internal representations of neural networks. By training a simple classifier (a probe) on frozen representations, researchers aim to determine if a model has captured specific linguistic properties, such as Part-of-Speech (POS) tags. However, a significant methodological risk exists: the Memorization Problem. A probe might achieve high accuracy not by identifying linguistic abstractions within the representation, but by simply memorizing the mapping between a word's identity and its label (e.g., learning that "apple" is a noun). This leads to "false positive" interpretations of a model's linguistic competence.

Methodology: Symmetric Selection

To address this, we introduce a Symmetric Selection method to quantify the gap between memorization and generalization. The process involves:

  1. Data Splitting: Randomly dividing training sentences into two equal halves: seen and unseen.
  2. Probe Training: Training the classifier exclusively on the "seen" sentences.
  3. Word Set Definition: Identifying "Seen Words" (appearing only in seen sentences) and "Unseen Words" (appearing only in unseen sentences). Words appearing in both are excluded to ensure a clean comparison.
  4. Quantification: The difference in accuracy between "seen words" and "unseen words" on a separate test set defines the Memorization Effect. A large gap indicates the probe is relying on rote memory of word identities rather than the features of the representation.

Experimental Results

We evaluated this method using a Transformer-based Neural Machine Translation (NMT) encoder trained on the CzEng dataset, with POS tagging as the probing task.

Accuracy and memorization gap (diff) for seen vs. unseen words across different probing setups and training set sizes.

The results, summarized in the table above, reveal several critical insights:

  • Static vs. Contextual Representations: Static word embeddings (Encoder word embeddings) exhibit a massive memorization gap, reaching up to 24.1 percentage points with small training sets. Even with 30,000 sentences, a significant gap remains. In contrast, contextual representations (Encoder output states) are much more resistant, with gaps never exceeding 5 percentage points.
  • Classifier Complexity: Using a Multi-Layer Perceptron (MLP) instead of a linear classifier generally increases the memorization effect. While MLPs often achieve higher raw accuracy, the "diff" column shows they are more prone to memorizing word identities.
  • The Reliability Estimate: For the linear classifier on contextual states with 30,000 training sentences, the memorization gap drops to 0.0. This suggests that in this specific configuration, the probe's performance is entirely attributable to linguistic generalization, providing a high reliability estimate for the probing results.

Discussion and Limitations

A notable characteristic of this method is its focus on lower-frequency words, as high-frequency words are likely to appear in both halves of the data split and thus be excluded. We argue this is advantageous: low-frequency words are precisely where a model is forced to rely on abstraction. However, the underrepresentation of medium-frequency and function words remains a limitation. Future work will explore applying this metric to diverse linguistic tasks and refining the sampling process to cover a broader frequency spectrum.

Conclusion

By explicitly measuring the memorization effect, researchers can choose probing setups that favor generalization over rote memory. This method provides a necessary sanity check, ensuring that the "linguistic knowledge" we attribute to neural networks is truly present in their representations and not just an artifact of the probing classifier's capacity to memorize.

Summary updated

Abstract

Multiple studies have probed representations emerging in neural networks trained for end-to-end NLP tasks and examined what word-level linguistic information may be encoded in the representations. In classical probing, a classifier is trained on the representations to extract the target linguistic information. However, there is a threat of the classifier simply memorizing the linguistic labels for individual words, instead of extracting the linguistic abstractions from the representations, thus reporting false positive results. While considerable efforts have been made to minimize the memorization problem, the task of actually measuring the amount of memorization happening in the classifier has been understudied so far. In our work, we propose a simple general method for measuring the memorization effect, based on a symmetric selection of comparable sets of test words seen versus unseen in training. Our method can be used to explicitly quantify the amount of memorization happening in a probing setup, so that an adequate setup can be chosen and the results of the probing can be interpreted with a reliability estimate. We exemplify this by showcasing our method on a case study of probing for part of speech in a trained neural machine translation encoder.
BibTeX
@inproceedings{rosa2020measuring,
  author = {Rudolf Rosa and Tomás Musil and David Marecek},
  title = {{Measuring Memorization Effect in Word-Level Neural Networks Probing}},
  booktitle = {TDS},
  year = {2020},
  url = {https://openreview.net/forum?id=-2Wq62bSCoQ},
}
Figure