United We Stand: Using Epoch-Wise Agreement of Ensembles to Combat Overfit

Uri Stern, Daniel Shwartz, Daphna Weinshall

TL;DR

Deep neural networks eventually "memorize" noisy data, causing test accuracy to plummet late in training. This paper introduces Max Agreement Prediction (MAP), a novel ensemble method that tracks prediction consistency across training epochs to filter out these late-stage errors, effectively reclaiming useful knowledge that standard "early stopping" techniques typically discard.

What Problem Does This Paper Address?

Overfitting is the "Achilles' heel" of deep learning. It occurs when a model becomes so fine-tuned to the specific quirks (and errors) of a training set that it loses its ability to generalize to new data. A major driver of overfit is label noise—incorrectly labeled data points that the network eventually memorizes.

The standard industry fix is early stopping: halting training the moment performance on a validation set starts to dip. However, the authors argue this is wasteful. Models often continue to learn useful, subtle features even while they are beginning to overfit. The challenge is: how do we extract that "late-stage" knowledge without being poisoned by the overfit?

Methods

The authors propose an ensemble-based approach called Max Agreement Prediction (MAP). While traditional ensembles usually just average the final predictions of several models, MAP looks at the entire history of the models' predictions.

  1. Theoretical Foundation: The authors prove (mathematically for linear models and empirically for deep ones) that while models tend to agree on correct labels throughout training, their "disagreement" spikes when they start overfitting to noise.
  2. The Agreement Score: For every test sample, MAP calculates how often the ensemble members agreed on a specific class across all training epochs.
  3. The MAP Rule: Instead of picking the class the models predict at the very last epoch, MAP selects the class that attained the highest "consensual prediction" throughout the training history.
  4. Simplicity: MAP is a post-processing step. It doesn't require changing the loss function or the training architecture; you simply save checkpoints during training and aggregate them at inference time.

Datasets

The researchers evaluated MAP across a diverse range of tasks to prove its robustness:

  • Image Classification (Synthetic Noise): CIFAR-10, CIFAR-100, TinyImageNet, and ImageNet-100 with varying levels of symmetric and asymmetric label noise.
  • Image Classification (Real-world Noise): Clothing1M, WebVision, and Animal10N (datasets known for having "natural" errors from web-scraping or human mistakes).
  • Natural Language Processing (NLP): Tasks from the GLUE benchmark, including MNLI (Inference), QNLI (Question Answering), and QQP (Question Pairs).

Key Insights and Ideas

  • Persistence is Truth: Correct labels are "persistent"—models tend to find them early and stick with them. Noisy labels are "transient"—different models in an ensemble will memorize the same noise at different times or in different ways, leading to low historical agreement.
  • Beyond Early Stopping: MAP often outperforms the "oracle" version of early stopping (the best possible single-epoch performance). This suggests that the "overfit phase" of training still contains valuable signal that MAP is able to harvest.
  • The "Double Descent" Connection: The paper notes that even when a single network's accuracy "rebounds" late in training (a phenomenon known as double descent), MAP still provides a more stable and higher-accuracy result than the raw ensemble.

Key Results

The quantitative gains are significant, particularly in high-noise environments:

  • CIFAR-10 (40% noise): MAP achieved 90.0% accuracy, compared to 67.7% for a single network and 78.5% for a standard "majority vote" ensemble.
  • Clothing1M (Real noise): MAP reached 71.7%, outperforming the single network baseline of 65.1%.
  • Stability: Unlike standard training curves that "peak and crash," the MAP accuracy curve is remarkably flat and stable. Once it reaches high accuracy, it stays there, removing the stress of perfectly timing the "stop" button.
  • Efficiency: The authors found that you don't need every epoch. Saving just a few equally spaced checkpoints (e.g., 10-20) is often enough to get near-optimal results.

Limitations and Future Work

  • Computational Overhead: Training an ensemble of 5+ networks is 5x more expensive than training one. While this is standard for high-stakes ML, it may be prohibitive for resource-constrained settings.
  • Storage: Saving multiple checkpoints for each model increases storage requirements, though the authors show that a small subset of checkpoints is usually sufficient.
  • Inference Latency: At test time, you must run the input through multiple models (or multiple checkpoints), which increases the time it takes to get a prediction.
  • Future Directions: The authors suggest exploring how to integrate MAP with other regularization techniques or finding ways to achieve similar "agreement" benefits using a single model's internal states rather than a full ensemble.

Summary updated

Abstract

Deep neural networks have become the method of choice for solving many image classification tasks, largely because they can fit very complex functions defined over raw images. The downside of such powerful learners is the danger of overfitting the training set, leading to poor generalization, which is usually avoided by regularization and "early stopping" of the training. In this paper, we propose a new deep network ensemble classifier that is very effective against overfit. We begin with the theoretical analysis of a regression model, whose predictions - that the variance among classifiers increases when overfit occurs - is demonstrated empirically in deep networks in common use. Guided by these results, we construct a new ensemble-based prediction method designed to combat overfit, where the prediction is determined by the most consensual prediction throughout the training. On multiple image and text classification datasets, we show that when regular ensembles suffer from overfit, our method eliminates the harmful reduction in generalization due to overfit, and often even surpasses the performance obtained by early stopping. Our method is easy to implement, and can be integrated with any training scheme and architecture, without additional prior knowledge beyond the training set. Accordingly, it is a practical and useful tool to overcome overfit.
BibTeX
@inproceedings{stern2024united,
  author = {Uri Stern and Daniel Shwartz and Daphna Weinshall},
  title = {{United We Stand: Using Epoch-Wise Agreement of Ensembles to Combat Overfit}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i13.29429},
}

Loading related content...

Figure