Let's Agree to Agree: Neural Networks Share Classification Order on Real Datasets

Guy Hacohen, Leshem Choshen, Daphna Weinshall

Abstract

We report a series of robust empirical observations, demonstrating that deep Neural Networks learn the examples in both the training and test sets in a similar order. This phenomenon is observed in all the commonly used benchmarks we evaluated, including many image classification benchmarks, and one text classification benchmark. While this phenomenon is strongest for models of the same architecture, it also crosses architectural boundaries -- models of different architectures start by learning the same examples, after which the more powerful model may continue to learn additional examples. We further show that this pattern of results reflects the interplay between the way neural networks learn benchmark datasets. Thus, when fixing the architecture, we show synthetic datasets where this pattern ceases to exist. When fixing the dataset, we show that other learning paradigms may learn the data in a different order. We hypothesize that our results reflect how neural networks discover structure in natural datasets.

Introduction

Training a deep neural network is a notoriously stochastic process. Between random weight initialization and the random shuffling of data in Stochastic Gradient Descent (SGD), two models with identical architectures can end up in very different regions of the optimization landscape. While it is well-known that these models often reach similar final accuracies, their internal trajectories have remained poorly understood. Do they all struggle with the same "hard" images? Do they all breeze through the same "easy" ones at the start of training?

This paper addresses these questions by demonstrating that neural networks are surprisingly consistent: they tend to learn the same examples in the same order on real-world datasets. This "universal learning order" suggests that the difficulty of a data point is not just subjective, but an inherent property of how neural architectures interact with the structure of natural data.

Quantifying Consensus: TP-Agreement

To measure how similarly models learn, the authors introduce a simple but powerful metric called TP-agreement (True-Positive agreement).

  • TP-agreement: For a collection of $N$ models trained on the same data, this is the fraction of models that classify a specific example correctly at a specific epoch.
  • Agreement Score: A broader version measuring the fraction of models that predict the same label, regardless of correctness.
  • Accessibility Score: The average TP-agreement of an example over the entire training process. A high score indicates an example was "easy" and learned early by nearly all models.
Illustration of TP-agreement and Agreement metrics.

By tracking these metrics across thousands of training runs, the authors could visualize whether models were "in sync." If models learned independently, the TP-agreement distribution would resemble a bell curve. However, if they were perfectly synchronized, the distribution would be bi-modal: for any given image, either 100% of models get it right, or 0% do.

The Bi-modal Phenomenon

On real-world datasets, the TP-agreement is strikingly bi-modal. At almost any point in training, models are in "consensus"—they either all know an example or all do not. There is very little disagreement where half the models get it right and half get it wrong.

TP-agreement distributions and heatmaps over training epochs.

As shown in the figure above, the distribution of data points shifts from being mostly unknown (TP-agreement near 0) to mostly known (TP-agreement near 1) as training progresses, with very few points lingering in the middle. This suggests that the "difficulty" of an example is shared across different initializations of the same architecture.

Generality Across Architectures and Tasks

One of the most significant findings is that this shared learning order persists across architectural boundaries. Stronger architectures (like ResNet-50) do not just learn "differently" than weaker ones (like AlexNet); they learn in a nested fashion. A ResNet starts by learning exactly what the AlexNet learns, in the same order, and only then moves on to the "harder" examples that the AlexNet never masters. The "Accessibility Scores" between different architectures show correlations as high as $r = 0.97$ to $0.99$.

This phenomenon is not limited to images. The study covered a broad range of benchmarks, including:

  • Image Classification: ImageNet, CIFAR-10/100, MNIST, and VGGFace2.
  • Text Classification: A dataset of 39k Stack Overflow questions.
  • Robustness: The order persists regardless of changes to hyperparameters like learning rate, batch size, or dropout.

Why Natural Data Matters

This synchronized learning order is not a universal property of neural networks on any data. It disappears on synthetic datasets (like Gabor patches) or when labels are randomized. This suggests that the shared learning order is a result of neural networks discovering specific, hierarchical structures present in "natural" data. Furthermore, when using a different learning paradigm like AdaBoost with linear classifiers, the learning order changed significantly, implying the order is a signature of the specific inductive bias of neural networks.

Conclusion

While primarily empirical, this work reveals a fundamental consistency in deep learning. The existence of a universal learning order suggests that the difficulty of natural data is objective and tied to the inductive biases of neural architectures. Future work could leverage these "accessibility scores" to design automated curriculum learning schedules or use Agreement scores as a tool for detecting out-of-distribution data and adversarial attacks.

Summary updated

Abstract

We report a series of robust empirical observations, demonstrating that deep Neural Networks learn the examples in both the training and test sets in a similar order. This phenomenon is observed in all the commonly used benchmarks we evaluated, including many image classification benchmarks, and one text classification benchmark. While this phenomenon is strongest for models of the same architecture, it also crosses architectural boundaries -- models of different architectures start by learning the same examples, after which the more powerful model may continue to learn additional examples. We further show that this pattern of results reflects the interplay between the way neural networks learn benchmark datasets. Thus, when fixing the architecture, we show synthetic datasets where this pattern ceases to exist. When fixing the dataset, we show that other learning paradigms may learn the data in a different order. We hypothesize that our results reflect how neural networks discover structure in natural datasets.
BibTeX
@inproceedings{hacohen2020let,
  author = {Guy Hacohen and Leshem Choshen and Daphna Weinshall},
  title = {{Let's Agree to Agree: Neural Networks Share Classification Order on Real Datasets}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=tZsopqYK2_3},
}

Loading related content...

Figure