Generalisation First, Memorisation Second? Memorisation Localisation for Natural Language Classification Tasks

Verna Dankers, Ivan Titov

Abstract

Memorisation is a natural part of learning from real-world data: neural models pick up on atypical input-output combinations and store those training examples in their parameter space. While this phenomenon is well-documented, the specific location of memorised information within a model's millions of parameters remains a subject of debate. In Computer Vision, a dominant hypothesis suggests that lower layers learn generalisable features while deeper layers specialise in memorisation. However, NLP research has provided conflicting evidence. This study expands the scope of the localisation question to 12 natural language classification tasks using four distinct localisation techniques. Our results indicate that memorisation is a gradual, task-dependent process rather than a strictly localised one, providing necessary nuance to the "generalisation first, memorisation second" hypothesis.

Introduction

Deep learning models possess a dual nature: they are remarkably adept at generalising patterns to new data, yet they are equally capable of memorising specific training examples, including noise and mislabelled data. Understanding where this memorisation occurs is critical for model interpretability and safety. If memorisation is localised, we might be able to "surgically" remove problematic data or biases.

In the realm of Computer Vision (CV), the prevailing theory is that models follow a "generalisation first, memorisation second" pipeline. Under this view, early layers act as feature extractors for broad patterns, while the final layers handle the exceptions and specificities. In Natural Language Processing (NLP), however, the evidence is less clear. This paper systematically investigates memorisation across a wide variety of NLP tasks to determine if the CV paradigm holds true for language.

Methodology

To study memorisation, the researchers employed a "noise memorisation" framework. They intentionally corrupted 15% of the training data by flipping their labels to incorrect ones. Because these labels are assigned randomly, the model cannot rely on generalisable linguistic patterns to predict them; it must memorise these specific instances to achieve high training accuracy.

Localisation Techniques

The study evaluated four Transformer-based models (BERT, OPT, Pythia, and GPT-Neo) using four complementary techniques to pinpoint memorised information:

  1. Layer Retraining: Specific layers were reset and retrained on clean data to observe if the memorised noise was eliminated.
  2. Layer Swapping: Layers were swapped between a model trained on noisy data and one trained on clean data to isolate the impact of specific parameters.
  3. Forgetting Gradients: The researchers measured which layers required the most significant updates when the model was forced to "unlearn" the noisy labels.
  4. Probing: Small classifiers were trained on the model's internal hidden states at each layer to see where the representation of a "clean" example first began to look like a "noisy" one.

To synthesize these findings, the authors introduced the Memorisation Centre-of-Gravity (M-CoG), a metric that quantifies whether memorisation is concentrated in the early, middle, or late layers of the architecture.

Experimental Setup

The investigation spanned 12 NLP classification tasks, categorized into four distinct groups:

  • Natural Language Understanding (NLU): WiC, RTE, MRPC, CoLA, BoolQ.
  • Sentiment Analysis: SST-2, SST-5, Emotion.
  • Hate Speech Detection: ImplicitHate, Stormfront.
  • Topic Classification: Reuters, TREC.

Key Insights

Memorisation as a Distributed Process

One of the most significant findings is that memorisation is a "team effort." Contrary to the idea of specific "knowledge neurons" or dedicated layers for exceptions, memorisation is a gradual process. Noisy examples do not suddenly switch to the incorrect class at a specific deep layer; instead, their internal representations slowly drift away from the correct class starting from the very first layers of the model.

The Role of Task Complexity

The location of memorisation is highly dependent on the task. NLU tasks, which require complex logical reasoning (like recognizing textual entailment), tend to begin the memorisation process in much earlier layers than simpler tasks like topic classification. This suggests that the more complex the linguistic processing required, the earlier the model must commit to a specific (even if incorrect) label.

Nuancing the Generalisation-Memorisation Trade-off

The researchers found a correlation between a model's ability to generalise and the depth of its memorisation. The better a model performs on clean data for a specific task, the more it tends to "push" memorisation toward the later layers. This provides a vital update to the "generalisation first" hypothesis: it is not a universal architectural rule, but rather a tendency that strengthens as model performance improves.

Early Intervention Effectiveness

Because the seeds of memorisation are planted early in the network, the study found that intervening in lower layers—through swapping or retraining—was significantly more effective at removing memorised noise than attempting to fix the final layers. Modifying a single layer was rarely sufficient; it typically required modifying 7 to 10 layers to fully revert the model's memorised behaviour.

Conclusion and Limitations

This research demonstrates that memorisation in NLP is a gradual, task-dependent phenomenon that challenges the simple "early generalisation, late memorisation" model. While the findings were consistent across different architectures (BERT vs. OPT) and scaled to larger models (OPT-1.3B), the authors note several limitations. The use of label flipping is a proxy for real-world memorisation, and the aggregate analysis of noisy examples might miss individual examples stored in highly localised "pockets."

Ultimately, the study suggests that the "where" of memorisation is inextricably linked to the "what" of the task, moving the field toward a more nuanced understanding of how Transformers balance general patterns with specific exceptions.

Summary updated

Abstract

Memorisation is a natural part of learning from real-world data: neural models pick up on atypical input-output combinations and store those training examples in their parameter space. That this happens is well-known, but how and where are questions that remain largely unanswered. Given a multi-layered neural model, where does memorisation occur in the millions of parameters? Related work reports conflicting findings: a dominant hypothesis based on image classification is that lower layers learn generalisable features and that deeper layers specialise and memorise. Work from NLP suggests this does not apply to language models, but has been mainly focused on memorisation of facts. We expand the scope of the localisation question to 12 natural language classification tasks and apply 4 memorisation localisation techniques. Our results indicate that memorisation is a gradual process rather than a localised one, establish that memorisation is task-dependent, and give nuance to the generalisation first, memorisation second hypothesis.
BibTeX
@inproceedings{dankers2024generalisation,
  author = {Verna Dankers and Ivan Titov},
  title = {{Generalisation First, Memorisation Second? Memorisation Localisation for Natural Language Classification Tasks}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=zm9I029C32},
}

Loading related content...

Figure