Generalisation First, Memorisation Second? Memorisation Localisation for Natural Language Classification Tasks

Verna Dankers, Ivan Titov

Abstract

Memorisation is a natural part of learning from real-world data: neural models pick up on atypical input-output combinations and store those training examples in their parameter space. That this happens is well-known, but how and where are questions that remain largely unanswered. Given a multi-layered neural model, where does memorisation occur in the millions of parameters? Related work reports conflicting findings: a dominant hypothesis based on image classification is that lower layers learn generalisable features and that deeper layers specialise and memorise. Work from NLP suggests this does not apply to language models, but has been mainly focused on memorisation of facts. We expand the scope of the localisation question to 12 natural language classification tasks and apply 4 memorisation localisation techniques. Our results indicate that memorisation is a gradual process rather than a localised one, establish that memorisation is task-dependent, and give nuance to the generalisation first, memorisation second hypothesis.

Introduction

Deep learning models are often described as "stochastic parrots" that simultaneously learn general rules (generalization) and store specific, often atypical, training examples (memorization). While memorization is necessary for learning facts, it also leads to overfitting and privacy risks. In Computer Vision (CV), the dominant hypothesis is "Generalisation First, Memorisation Second"—meaning lower layers extract general features while deeper layers handle the "brute force" memorization of outliers. However, evidence in Natural Language Processing (NLP) has been conflicting. This study investigates where memorization happens in a transformer's layers for text classification, asking whether it is localized to specific "memory" layers or is a network-wide effort.

Experimental Setup

The authors employed a controlled experimental setup using label perturbation. They took 12 datasets and intentionally flipped the labels of 15% of the training data ("noisy" examples), forcing the model to memorize these incorrect associations to achieve high training accuracy.

To locate where this memorization occurs, four localization techniques were used:

  1. Layer Retraining: Resetting specific layers and retraining them only on clean data.
  2. Layer Swapping: Replacing layers of a "memorized" model with layers from a model trained on clean data.
  3. Forgetting Gradients: Measuring which layers require the largest updates to "unlearn" the noisy labels.
  4. Probing: Training small classifiers on the hidden states of each layer to see if the model "knows" an example is noisy.

The study evaluated four 12-layer models (BERT, OPT, Pythia, and GPT-Neo) across 12 diverse tasks, including NLU (e.g., RTE, MRPC), Sentiment (e.g., SST-2), Hate Speech detection, and Topic Classification.

Results and Analysis

A key finding is that memorization is not localized. Modifying just one layer had a negligible effect on memorization error (usually <15%). To fully "erase" memorized noise, the authors had to modify 7 to 10 layers, suggesting a network-wide process.

Task-dependent memorization patterns: NLU tasks (top) show memorization starting earlier, while topic classification (bottom) follows the 'generalization first' pattern more closely.

The study identified an "NLU Exception". For NLU tasks like entailment (RTE), memorization starts significantly earlier in the network compared to topic classification. In NLU, the model's internal representations of "noisy" examples start drifting toward the wrong class almost immediately. The "Generalisation First" hypothesis is thus task-dependent: the better a model is at generalizing a specific task, the more it "pushes" memorization into the deeper layers.

Key Results

The authors introduced the M-CoG (Memorisation Centre-of-Gravity) metric to summarize the "depth" of memorization. NLU tasks consistently showed a lower (shallower) M-CoG than topic or sentiment tasks. This pattern was consistent across different models; despite varying pre-training data, BERT, OPT, and GPT-Neo showed remarkably similar task-based memorization depths (Spearman’s $\rho \approx 0.85$).

Summary of localization accuracy across methods, tasks, and models. The results show consistent patterns across different architectures and the effectiveness of intervention methods like swapping and retraining.

Intervention success further supported these findings: swapping the bottom six layers was far more effective at reverting memorization than swapping the top six layers, proving that the foundations of memorization are laid early in the forward pass for many tasks.

Conclusion

This research indicates that memorization in transformers is a gradual, cooperative process rather than a localized one. While the "Generalisation First" hypothesis holds for some tasks, complex NLU tasks often require memorization to begin in the earliest layers.

Limitations include the use of artificial label noise, which may differ from real-world memorization of rare linguistic exceptions. Additionally, while the patterns held for a 1.3B parameter model, the dynamics in massive 70B+ parameter models remain an open question. Future work should explore the specific task properties, such as syntactic complexity, that drive early-layer memorization.

Summary updated

Abstract

Memorisation is a natural part of learning from real-world data: neural models pick up on atypical input-output combinations and store those training examples in their parameter space. That this happens is well-known, but how and where are questions that remain largely unanswered. Given a multi-layered neural model, where does memorisation occur in the millions of parameters?Related work reports conflicting findings: a dominant hypothesis based on image classification is that lower layers learn generalisable features and that deeper layers specialise and memorise. Work from NLP suggests this does not apply to language models, but has been mainly focused on memorisation of facts.We expand the scope of the localisation question to 12 natural language classification tasks and apply 4 memorisation localisation techniques.Our results indicate that memorisation is a gradual process rather than a localised one, establish that memorisation is task-dependent, and give nuance to the generalisation first, memorisation second hypothesis.
BibTeX
@inproceedings{dankers2024generalisation,
  author = {Verna Dankers and Ivan Titov},
  title = {{Generalisation First, Memorisation Second? Memorisation Localisation for Natural Language Classification Tasks}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=zm9I029C32},
}

Loading related content...

Figure