TL;DR
Probing classifiers used to interpret neural networks often "cheat" by memorizing word-to-label mappings instead of identifying linguistic abstractions. This paper introduces a symmetric sampling method to quantify this "memorization effect," revealing that while contextual embeddings generalize well, static embeddings and complex probe classifiers are highly prone to memorizing identities.
What Problem Does This Paper Address?
In the world of Natural Language Processing (NLP), "probing" is a standard diagnostic tool. Researchers train a simple classifier (a probe) on top of frozen neural network representations (like BERT embeddings) to see if those representations "know" linguistic properties like Part-of-Speech (POS) tags.
The danger is the Memorization Problem: Does the probe succeed because the representation contains a high-level linguistic abstraction, or did the probe simply learn that the word "apple" is usually a noun? If the probe just memorizes word identities, it produces a "false positive," leading researchers to believe a model is more linguistically aware than it actually is. While splitting vocabularies works for static embeddings, it’s difficult for contextual embeddings because words appear in many different sentence contexts, making a clean "seen vs. unseen" split tricky to balance.
Methods
The authors propose a Symmetric Selection method to explicitly measure the gap between memorization and generalization:
- Split the Data: Randomly split the training sentences into two equal halves: seen sentences and unseen sentences.
- Train the Probe: Train the probing classifier only on the "seen" sentences.
- Define Word Sets:
- Seen Words: Words that appear in the "seen" sentences but never in the "unseen" sentences.
- Unseen Words: Words that appear in the "unseen" sentences but never in the "seen" sentences.
- (Words appearing in both are discarded to ensure a clean comparison).
- Quantify Memorization: Evaluate the probe on a separate test set. The difference in accuracy between "seen words" and "unseen words" is the Memorization Effect.
If the accuracies are identical, the probe is generalizing based on the representation's features. If the "seen" accuracy is much higher, the probe is relying on memorized word identities.
Datasets
- Model Training: A Transformer-based Neural Machine Translation (NMT) model trained on the CzEng dataset (Czech-English parallel corpus).
- Probing Task: Part-of-Speech (POS) tagging using the Czech Prague Dependency Treebank (PDT) 1.4.
- Vocabulary: A fixed vocabulary of the 25,000 most frequent words.
Key Insights and Ideas
- The Frequency Bias: This method naturally focuses on lower-frequency words. Frequent words (like "the" or "and") appear in almost every sentence and are thus filtered out of the "seen/unseen" sets. The authors argue this is a feature, not a bug: low-frequency words are exactly where the network is forced to rely on abstraction rather than rote memory.
- Static vs. Contextual: Static word embeddings (where one word always has one vector) are massive "cheats." They encode word identity so strongly that probes almost always memorize the label. Contextual embeddings (where "bank" changes based on context) are much more resistant to this.
- Classifier Complexity Matters: Using a Multi-Layer Perceptron (MLP) as a probe instead of a simple Linear classifier increases the risk of memorization. A "stronger" probe isn't always better; it might just be a better memorizer.
Key Results
- Static Embeddings: Showed a massive memorization gap, ranging from 4 to 24 percentage points. Even with a lot of data, the probe still relied heavily on knowing which word was which.
- Contextual Representations: The memorization effect was much smaller, never surpassing 5 percentage points.
- The "Zero Gap" Sweet Spot: When using a linear classifier on contextual encoder states with 30,000 training sentences, the memorization gap dropped to 0.0. This provides a "reliability estimate," suggesting that in this specific setup, the probe's results are 100% due to linguistic generalization.
Limitations and Future Work
The primary limitation is the underrepresentation of medium-frequency words. Because these words often appear in both halves of the data split, they are excluded from the measurement. The authors also note that very frequent "function words" (prepositions, pronouns) are difficult to evaluate with this method, though they argue these words are often treated as special cases by neural networks anyway.
Future work aims to apply this metric to a wider range of tasks beyond POS tagging and to refine the sampling process to include a broader spectrum of word frequencies.
Summary updated