Relearning Forgotten Knowledge: on Forgetting, Overfit and Training-Free Ensembles of DNNs

Uri Stern, Daphna Weinshall

Abstract

The infrequent occurrence of overfit in deep neural networks is perplexing. On the one hand, theory predicts that as models get larger they should eventually become too specialized for a specific training set, with ensuing decrease in generalization. In contrast, empirical results in image classification indicate that increasing the training time of deep models or using bigger models almost never hurts generalization. Is it because the way we measure overfit is too limited? Here, we introduce a novel score for quantifying overfit, which monitors the forgetting rate of deep models on validation data. Presumably, this score indicates that even while generalization improves overall, there are certain regions of the data space where it deteriorates. When thus measured, we show that overfit can occur with and without a decrease in validation accuracy, and may be more common than previously appreciated. This observation may help to clarify the aforementioned confusing picture. We use our observations to construct a new ensemble method, based solely on the training history of a single network, which provides significant improvement in performance without any additional cost in training time. An extensive empirical evaluation with modern deep models shows our method's utility on multiple datasets, neural networks architectures and training schemes, both when training from scratch and when using pre-trained networks in transfer learning. Notably, our method outperforms comparable methods while being easier to implement and use, and further improves the performance of competitive networks on Imagenet by 1%.

Introduction: The Problem of Hidden Overfitting

In classical machine learning theory, there is a clear trade-off: as a model becomes more complex or trains longer, it eventually overfits the training data, causing its performance on new data (generalization) to plummet. However, modern deep learning often defies this. Huge models frequently show "benign overfitting" or even "double descent," where performance keeps improving despite the model having enough capacity to memorize the entire training set.

The authors argue that standard global test accuracy is too blunt a metric. It hides "local" overfitting, where a model improves on average while becoming worse at specific sub-regions of the data. This "hidden" forgetting suggests that the final version of a model isn't necessarily the best version for every single data point.

Quantifying Forgetting: The Forget Fraction ($F_e$)

To diagnose this, the researchers introduce the Forget Fraction ($F_e$). This metric tracks the portion of test data that a model classifies correctly at an intermediate epoch ($e$) but misclassifies by the end of training ($E$). If $F_e$ is high, the model has "forgotten" knowledge it previously possessed.

The Forget Fraction ($F_e$) across different settings. (a) $F_e$ over training epochs for TinyImageNet. (b) Distribution of mistakes relative to their correct prediction duration. (c) Comparison of forget rates across ConvNeXt model sizes on ImageNet. (d) Forget rates for varying data percentages on CIFAR-100.

As shown in Figure 6, forgetting is universal even on clean datasets. Paradoxically, larger and more powerful models (e.g., ConvNeXt-Large vs. ConvNeXt-Small) tend to have higher forget fractions (panel c), suggesting that the need for knowledge recovery increases with model scale.

Knowledge Fusion (KF)

The authors propose Knowledge Fusion (KF), a "training-free" ensemble method to recover this lost knowledge. KF identifies epochs where forgetting was highest using a validation set and creates a weighted average of class probabilities from the final model and these specific mid-training checkpoints. Unlike other ensemble methods like Stochastic Weight Averaging (SWA), KF doesn't require changing the training schedule; one simply saves checkpoints during standard training and combines them afterward.

Dynamics in Noisy Environments

In datasets with noisy labels, models often exhibit a "double descent" in error. The authors found that the second period of improvement isn't the model "re-learning" what it forgot during the overfit phase; rather, it's learning entirely new, general patterns while the forgotten specific patterns stay lost.

Learning dynamics on noisy datasets. Panels (a) and (b) show test accuracy vs. examples correctly predicted after training. Panels (c) and (d) illustrate the difference between clean and noisy data amounts, highlighting phases where clean labels (positive) or noisy labels (negative) dominate learning.

Figure 2 illustrates these dynamics, showing how clean labels dominate early learning before noisy labels begin to interfere, leading to the characteristic accuracy fluctuations seen in noisy benchmarks like CIFAR-100 and TinyImageNet.

Key Results and Performance

The KF method was validated across standard benchmarks (CIFAR-100, TinyImageNet, ImageNet-1K) and noisy datasets (Animal10N). Key findings include:

  • ImageNet: A ~1% absolute improvement in accuracy across architectures like ResNet50, ConvNeXt, and ViT.
  • Noisy Data: On CIFAR-100 with 40% asymmetric noise, KF improved accuracy by 12.6% over the base model.
  • Sub-optimal Training: KF effectively "closes the gap" between poorly tuned models and optimized ones.
  • Fairness: The method improves accuracy without negatively impacting model fairness (amplification bias).

Conclusion

By identifying and re-integrating knowledge from "forgotten" mid-training states, Knowledge Fusion provides a significant performance boost across various architectures and training schemes. This work demonstrates that even when global accuracy suggests a model is improving, local forgetting is a prevalent issue that can be mitigated through simple, training-free ensembling of historical checkpoints.

Summary updated

Abstract

The infrequent occurrence of overfit in deep neural networks is perplexing. On the one hand, theory predicts that as models get larger they should eventually become too specialized for a specific training set, with ensuing decrease in generalization. In contrast, empirical results in image classification indicate that increasing the training time of deep models or using bigger models almost never hurts generalization. Is it because the way we measure overfit is too limited? Here, we introduce a novel score for quantifying overfit, which monitors the forgetting rate of deep models on validation data. Presumably, this score indicates that even while generalization improves overall, there are certain regions of the data space where it deteriorates. When thus measured, we show that overfit can occur with and without a decrease in validation accuracy, and may be more common than previously appreciated. This observation may help to clarify the aforementioned confusing picture. We use our observations to construct a new ensemble method, based solely on the training history of a single network, which provides significant improvement in performance without any additional cost in training time. An extensive empirical evaluation with modern deep models shows our method's utility on multiple datasets, neural networks architectures and training schemes, both when training from scratch and when using pre-trained networks in transfer learning. Notably, our method outperforms comparable methods while being easier to implement and use, and further improves the performance of competitive networks on Imagenet by 1\%.
BibTeX
@misc{stern2023relearning,
  author = {Uri Stern and Daphna Weinshall},
  title = {{Relearning Forgotten Knowledge: on Forgetting, Overfit and Training-Free Ensembles of DNNs}},
  year = {2023},
  eprint = {2310.11094},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.11094},
}
Figure