Despite the inherent randomness of neural network training—from random weight initializations to the stochastic nature of SGD—different models tend to learn the same examples in the exact same order. This research demonstrates that for natural datasets, there is a universal "difficulty hierarchy" that neural networks follow, regardless of their specific architecture or hyperparameters.
The Problem: Do Models Learn Data in the Same Way?
Training a deep neural network (DNN) is a notoriously "messy" process. Because the optimization landscape is highly non-convex, two models with the same architecture starting from different random seeds will end up with drastically different weights.
Researchers have long wondered: are these models truly different, or are they just taking different paths to the exact same functional result? While we know they often achieve similar final accuracy, this paper asks a deeper question: Do they learn the data in the same way? By investigating the "classification order"—the sequence in which specific data points are mastered during training—the authors seek to understand if there is an underlying structure in real-world data that dictates how all neural networks must learn.
Quantifying Learning Order
To measure how similarly models learn, the authors introduce two primary metrics to quantify model agreement throughout the learning process:
- $Tp\text{-agreement}$ (True-Positive Agreement): For a specific image and a specific epoch, what fraction of a population of models (e.g., 27 different ResNets) correctly classifies that image?
- Accessibility Score: This is the $Tp\text{-agreement}$ averaged over the entire training duration. An image with a high Accessibility score is learned early and remains correctly classified, while a low score indicates an image learned late or not at all.
By calculating these scores across thousands of models and various datasets, the authors could compare the "learning trajectories" of different architectures. They also used a "null hypothesis" baseline—comparing real models against random classification vectors with the same accuracy—to prove that the observed agreement wasn't just a statistical fluke of increasing accuracy.
Key Insights and Discoveries
The study is impressively broad, covering multiple domains including image classification (ImageNet, CIFAR-10/100, MNIST, etc.), text classification (Stack Overflow dataset), and synthetic/control datasets. Across these domains, several striking patterns emerged.
The Bi-modal Phenomenon
The most striking finding is that $Tp\text{-agreement}$ is almost always bi-modal (concentrated at 0 or 1). At any given point in training, for almost any image, either all models get it right or all models get it wrong. There is very little "middle ground" where models disagree on specific samples.
The Architecture Hierarchy
Stronger models (like ResNet) don't learn "differently" than weaker models (like AlexNet). Instead, they learn the exact same "easy" examples first, and then the stronger model continues to learn "harder" examples that the weaker model never masters.
- Near-Perfect Correlation: When comparing two different sets of ResNet-50 models on ImageNet, the correlation between their Accessibility scores was a staggering $r = 0.99$.
- Cross-Architecture Consistency: Even when comparing vastly different architectures like ResNet-50 and AlexNet, the correlation remained high at $r = 0.87$.
Natural vs. Synthetic Data
This shared learning order is a property of natural data. When models are trained on synthetic data (like Gabor patches) or random labels, the bi-modal agreement vanishes. Each model learns the random noise in its own unique, independent order.
Furthermore, the authors compared NNs to AdaBoost (a non-neural learning algorithm). AdaBoost learned the data in a significantly different order, suggesting that the "classification order" is a unique signature of how neural networks specifically perceive data structure.
Robustness to Hyperparameters
This learning order was found to be invariant to changes in learning rate, batch size, weight decay, and dropout. If you change the "how" of training, the "what" and "when" stay exactly the same.
Limitations and Future Work
While the paper proves that a shared order exists, the "why" remains elusive. The study doesn't define the specific image features (e.g., edges, textures, frequency) that make an image "easy" or "hard" for a network.
Interestingly, the authors briefly mention that using these Accessibility scores to create a "curriculum" (presenting easy examples before hard ones) did not significantly improve training results in their initial tests. This suggests that NNs might already be performing a sort of "implicit curriculum learning."
Finally, because the shared order disappears when models are forced to memorize random labels, future work could explore using $Tp\text{-agreement}$ as a diagnostic tool to detect when a model has stopped generalizing and started memorizing.
Summary updated