Abstract
Highly overparametrized neural networks can display curiously strong generalization performance - a phenomenon that has recently garnered a wealth of theoretical and empirical research in order to better understand it. In contrast to most previous work, which typically considers the performance as a function of the model size, in this paper we empirically study the generalization performance as the size of the training set varies over multiple orders of magnitude. These systematic experiments lead to some interesting and potentially very useful observations; perhaps most notably that training on smaller subsets of the data can lead to more reliable model selection decisions whilst simultaneously enjoying smaller computational costs. Our experiments furthermore allow us to estimate Minimum Description Lengths for common datasets given modern neural network architectures, thereby paving the way for principled model selection taking into account Occams-razor.
Introduction
In the current "bigger is better" era of deep learning, model selection and architecture search have become prohibitively expensive. Standard practice suggests that to determine which model performs best, one must train candidates on the full dataset. However, classical learning theory suggests that overparameterized models should overfit wildly on small datasets, making their performance metrics noisy and unreliable. This research addresses two main challenges: the computational burden of full-scale training and the instability of standard metrics like cross-entropy in the small-data regime.
Methodology: Taming the Loss Function
The researchers conducted an exhaustive empirical study across standard benchmarks—MNIST, EMNIST, CIFAR10, and ImageNet—systematically subsampled to create "small-data" versions. They trained a vast array of architectures, including MLPs, ConvNets, and ResNets, varying the training set size from the full dataset down to just a few examples per class.
The core of their approach is Temperature Calibration. By tuning a single scalar (the softmax temperature) on a small held-out calibration set, they stabilize the cross-entropy loss. This prevents the model from becoming overconfident, ensuring that cross-entropy remains a stable metric that correlates strongly with actual error rates, even when the model is massively overparameterized for the available data.
Key Insights and the Ranking Hypothesis
The paper's central claim is the Ranking Hypothesis: if Model A outperforms Model B on a small subset of data, it will almost certainly outperform it on the full dataset, provided both models are beyond the interpolation threshold. Surprisingly, the model size required to reach diminishing returns seems independent of the training set size; for instance, a ResNet-101 needs roughly the same number of channels to perform optimally whether trained on 10,000 or 1.2 million samples.
Furthermore, the authors demonstrate that calibrated cross-entropy is a near-perfect proxy for generalization error. This allows for reliable model selection at a fraction of the usual cost. On ImageNet, training on a 10% subset (128k samples) provided a robust signal while being 4x to 7x faster than full-set training.
Conclusion and Future Work
Model selection on small subsets is not merely a "good enough" approximation; it is often a more efficient and reliable signal. By applying temperature calibration, researchers can achieve significant speedups in architecture search. While the empirical evidence is strong across many architectures, the work is primarily empirical. A formal theoretical framework explaining why these rankings remain so stable across scales remains an open challenge. Additionally, the Ranking Hypothesis primarily holds in the overparameterized regime, and its applicability to very small models or specific hyperparameter interactions requires further exploration.
Summary updated