Exploring the Memorization-Generalization Continuum in Deep Learning

Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, Michael C. Mozer

Abstract

Humans are accustomed to environments that contain both regularities and exceptions. For example, at most gas stations, one pays prior to pumping, but the occasional rural station does not accept payment in advance. Likewise, deep neural networks can generalize across instances that share common patterns or structures, yet have the capacity to memorize rare or irregular forms. We analyze how individual instances are treated by a model via a consistency score. The score characterizes the expected accuracy for a held-out instance given training sets of varying size sampled from the data distribution. We obtain empirical estimates of this score for individual instances in multiple data sets, and we show that the score identifies out-of-distribution and mislabeled examples at one end of the continuum and strongly regular examples at the other end. We identify computationally inexpensive proxies to the consistency score using statistics collected during training. We show examples of potential applications to the analysis of deep-learning systems.

Introduction

Deep neural networks (DNNs) exhibit a dual nature: they are remarkably effective at extracting generalizable patterns from data, yet they possess the capacity to memorize virtually any training set, including random noise. This tension between generalization and memorization is central to deep learning, but the boundary between "regular" data (which supports generalization) and "irregular" data (which requires memorization) has historically been poorly defined.

In a standard dataset like ImageNet, "regular" examples—such as a clear, centered photograph of a golden retriever—are easily classified by models trained on small subsets of the data. Conversely, "irregular" examples—such as images where the subject is obscured, atypical, or incorrectly labeled—require the model to see a much larger portion of the dataset to achieve correct classification. This paper introduces the Consistency Score (C-score) to formally quantify this spectrum of structural regularity.

Quantifying Regularity: The C-Score

The authors propose a metric based on the Consistency Profile, which tracks the expected accuracy of a model on a specific held-out instance $(x, y)$ as the training set size $n$ increases. The C-score is the expectation of this profile over the distribution of training set sizes. Intuitively, a high C-score indicates an instance that is "easy" for the model to learn because it aligns with the core regularities of the data distribution.

Conceptual framework of the C-score showing consistency profiles (b) and visual examples from ImageNet sorted by score (c).

While calculating the true C-score is computationally intensive—requiring thousands of training runs on varying data subsets—the authors identify highly effective proxies. Specifically, they found that learning speed (how early and consistently a model correctly classifies a point during a single training run) correlates strongly ($\rho \approx 0.9$) with the full C-score. This allows researchers to estimate regularity "for free" during standard training.

The Regularity Continuum and Model Behavior

The C-score reveals that data exists on a smooth continuum rather than in binary categories of "in-distribution" or "outlier."

SGD’s Implicit Curriculum

A key insight from this work is that Stochastic Gradient Descent (SGD) naturally follows an "implicit curriculum." Early in training, SGD focuses on high C-score (regular) examples, building a foundational representation of the data's primary structures. Only later in training does the model begin to incorporate low C-score (irregular) examples.

Adam vs. SGD

This framework helps explain the performance gap often observed between Adam and SGD. While Adam converges faster, it does so by forcing the model to learn examples across the C-score spectrum at a more uniform pace. By bypassing the natural easy-to-hard curriculum of SGD, Adam may fail to develop as robust a representation of the data's core regularities, leading to slightly worse generalization on held-out data.

Applications and Results

The C-score provides a powerful tool for dataset diagnostics and optimization.

Outlier and Mislabeled Data Detection

The C-score is exceptionally effective at identifying problematic data points. By examining instances at the lowest end of the C-score spectrum, the authors could identify over 95% of label-flipped outliers in the CIFAR-10 dataset. These low-score points typically consist of mislabeled images, ambiguous cases (where multiple labels might apply), or highly atypical forms of a class.

Examples of low C-score instances across MNIST, CIFAR-10, and CIFAR-100, categorized as mislabeled, ambiguous, or atypical.

Dataset Pruning

The authors also explored dataset pruning using C-scores. On the SVHN dataset, removing the lowest-scoring examples—the "junk" and extreme outliers—actually improved model performance. This suggests that low C-score data can act as noise that hinders the learning of generalizable features. However, there is a limit: removing too many points eventually removes "rare but valid" examples, which harms the model's ability to handle edge cases.

Conclusion

The Consistency Score offers a rigorous way to map the structural regularities of large-scale datasets. By placing individual instances on a memorization-generalization continuum, it provides a new lens through which to view optimizer behavior, dataset quality, and the learning process itself. Future work will involve extending these metrics to non-classification tasks and exploring the regularity profiles of emerging architectures like Vision Transformers.

Summary updated

Abstract

Humans are accustomed to environments that contain both regularities and exceptions. For example, at most gas stations, one pays prior to pumping, but the occasional rural station does not accept payment in advance. Likewise, deep neural networks can generalize across instances that share common patterns or structures, yet have the capacity to memorize rare or irregular forms. We analyze how individual instances are treated by a model via a consistency score. The score characterizes the expected accuracy for a held-out instance given training sets of varying size sampled from the data distribution. We obtain empirical estimates of this score for individual instances in multiple data sets, and we show that the score identifies out-of-distribution and mislabeled examples at one end of the continuum and strongly regular examples at the other end. We identify computationally inexpensive proxies to the consistency score using statistics collected during training. We show examples of potential applications to the analysis of deep-learning systems.
BibTeX
@inproceedings{jiang2021exploring,
  author = {Ziheng Jiang and Chiyuan Zhang and Kunal Talwar and Michael C. Mozer},
  title = {{Exploring the Memorization-Generalization Continuum in Deep Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=LYtzVihRte},
}
Figure