Abstract
Deep neural networks have become the method of choice for solving many image classification tasks, largely because they can fit very complex functions defined over raw images. The downside of such powerful learners is the danger of overfitting the training set, leading to poor generalization, which is usually avoided by regularization and "early stopping" of the training. In this paper, we propose a new deep network ensemble classifier that is very effective against overfit. We begin with the theoretical analysis of a regression model, whose predictions - that the variance among classifiers increases when overfit occurs - is demonstrated empirically in deep networks in common use. Guided by these results, we construct a new ensemble-based prediction method designed to combat overfit, where the prediction is determined by the most consensual prediction throughout the training. On multiple image and text classification datasets, we show that when regular ensembles suffer from overfit, our method eliminates the harmful reduction in generalization due to overfit, and often even surpasses the performance obtained by early stopping. Our method is easy to implement, and can be integrated with any training scheme and architecture, without additional prior knowledge beyond the training set. Accordingly, it is a practical and useful tool to overcome overfit.
Introduction
Overfitting is often described as the "Achilles' heel" of deep learning. It occurs when a model becomes so fine-tuned to the specific quirks and errors of a training set—often driven by label noise—that it loses its ability to generalize to new data. The standard industry fix is early stopping: halting training the moment performance on a validation set starts to dip. However, this approach can be wasteful. Models often continue to learn useful, subtle features even while they are beginning to overfit. The challenge lies in extracting that late-stage knowledge without being poisoned by the overfit.
Methodology: Max Agreement Prediction (MAP)
We propose an ensemble-based approach called Max Agreement Prediction (MAP). Unlike traditional ensembles that typically average the final predictions of several models, MAP analyzes the entire history of the models' predictions across training epochs.
The Persistence of Truth
Our method is founded on the observation that correct labels are "persistent." Models tend to identify correct patterns early and stick with them. In contrast, noisy labels are "transient"; different models in an ensemble will memorize the same noise at different times or in different ways, leading to low historical agreement. We proved mathematically for linear models and demonstrated empirically for deep networks that while models agree on correct labels throughout training, their disagreement spikes when they start overfitting to noise.
The MAP Rule
For every test sample, MAP calculates an agreement score: how often the ensemble members agreed on a specific class across all training epochs. Instead of picking the class predicted at the very last epoch, MAP selects the class that attained the highest consensual prediction throughout the training history.
Experimental Evaluation
We evaluated MAP across a diverse range of tasks to prove its robustness:
- Image Classification: CIFAR-10/100, TinyImageNet, and ImageNet-100 with varying levels of synthetic noise, as well as real-world noisy datasets like Clothing1M and WebVision.
- Natural Language Processing: Tasks from the GLUE benchmark, including MNLI, QNLI, and QQP.
Key Results
The quantitative gains are significant, particularly in high-noise environments. On CIFAR-10 with 40% noise, MAP achieved 90.0% accuracy, compared to 67.7% for a single network and 78.5% for a standard majority-vote ensemble. Notably, MAP often outperforms the "oracle" version of early stopping (the best possible single-epoch performance), suggesting it successfully harvests signal from the overfit phase that other methods discard.
Discussion and Limitations
MAP is a post-processing step that is easy to implement and requires no changes to loss functions or architectures. It offers remarkable stability; unlike standard training curves that "peak and crash," the MAP accuracy curve remains flat once it reaches high accuracy, removing the stress of perfectly timing the "stop" button.
However, there are trade-offs. Training an ensemble is computationally more expensive than training a single model, and saving multiple checkpoints increases storage requirements. Future work will explore achieving similar agreement benefits using a single model's internal states to reduce these overheads.
Summary updated