Abstract
We report a series of robust empirical observations, demonstrating that deep Neural Networks learn the examples in both the training and test sets in a similar order. This phenomenon is observed in all the commonly used benchmarks we evaluated, including many image classification benchmarks, and one text classification benchmark. While this phenomenon is strongest for models of the same architecture, it also crosses architectural boundaries -- models of different architectures start by learning the same examples, after which the more powerful model may continue to learn additional examples. We further show that this pattern of results reflects the interplay between the way neural networks learn benchmark datasets. Thus, when fixing the architecture, we show synthetic datasets where this pattern ceases to exist. When fixing the dataset, we show that other learning paradigms may learn the data in a different order. We hypothesize that our results reflect how neural networks discover structure in natural datasets.
Introduction
Deep neural networks (DNNs) are often characterized as "black boxes." Despite their widespread success, the internal mechanics of how they arrive at a solution remain opaque. Because DNNs are typically initialized with random weights and trained using Stochastic Gradient Descent (SGD)—a process involving random data sampling—two identical models trained on the same dataset will result in entirely different internal weight configurations.
While these models often converge to similar final accuracy levels, a fundamental question remains: do they follow the same trajectory during training? Specifically, do different networks find the same examples "easy" or "hard," and do they learn them in a consistent sequence? This research investigates whether neural networks discover a universal structure in natural data or if their success is merely a collection of idiosyncratic, model-specific shortcuts.
Measuring the Learning Path
To quantify model behavior over time, the researchers introduced two novel metrics:
- TP-agreement (True-Positive Agreement): This metric measures the fraction of models in a large population that correctly classify a specific image at a given point in training.
- Accessibility Score: This represents the average TP-agreement of an image throughout the entire training process. A high score indicates an example that is learned early and remains "learned," while a low score denotes an example learned only very late or not at all.
By training hundreds of models—including ResNets, VGGs, and LSTMs—and comparing these scores, the authors could visualize a "consensus" among networks. This was compared against a null hypothesis to determine if the observed agreement exceeded what would be expected by random chance.
The Bi-modal Phenomenon and Universality
A key finding of the study is the "bi-modal" nature of TP-agreement on real-world datasets. At any given epoch, an image is typically either correctly classified by almost all models or almost none of them. There is remarkably little middle ground where models disagree on whether an example has been "learned."
Furthermore, this learning order appears to be universal across architectures. For instance, the correlation between the accessibility scores of a ResNet-50 and an AlexNet is remarkably high ($r=0.87$). This suggests that stronger models do not necessarily learn differently; rather, they learn everything a weaker model learns first, before proceeding to master more complex examples.
Generalization vs. Memorization
The researchers tested whether this shared order is a byproduct of the optimization algorithm (SGD) or a reflection of data structure. When labels were randomly shuffled—forcing models to memorize rather than generalize—the shared learning order vanished. This indicates that the consistent "curriculum" observed in training is a hallmark of the generalization process.
Interestingly, when using non-neural algorithms like AdaBoost, the learning order differed significantly. This implies that the observed "curriculum" is a specific result of how the neural network paradigm interacts with the inherent structure of natural data.
Conclusion and Future Directions
The study robustly demonstrates that all neural networks, regardless of initialization or architecture, tend to learn natural data in a similar order. While the exact reason why certain images are "easier" remains an open question, the authors hypothesize it relates to underlying data complexity.
These findings have significant implications for Curriculum Learning, suggesting that we can design better training schedules by following the "natural" order discovered by these networks. Additionally, the "Agreement Score" could serve as a powerful tool for novelty detection, identifying data points that deviate from the structural patterns that neural networks universally agree upon.
Summary updated