Reducing Flipping Errors in Deep Neural Networks

Xiang Deng, Yun Xiao, Bo Long, Zhongfei Zhang

Introduction

Standard training of deep neural networks (DNNs) is often assumed to be a progressive improvement toward an optimal state. However, empirical evidence suggests that models frequently "forget" correct predictions made during earlier stages of training. Researchers from SUNY Binghamton and JD.com identified this phenomenon as Flipping Errors—instances where a sample correctly classified in a previous epoch is misclassified in the final model state. To address this, they proposed Flipping Error Reduction (FER), a regularization technique that anchors a model's correct behaviors without increasing inference latency or parameter count.

The Flipping Error Phenomenon

The researchers introduced the Relative Flipping Error (RFE) metric to quantify the extent of this issue. RFE measures the proportion of final misclassifications that were correctly predicted at least once during the training process.

Relative Flipping Error (RFE) across different architectures and datasets.

As illustrated in Figure 2, RFEs are remarkably high—often exceeding 90% for architectures like ResNet-18 on CIFAR-100. This suggests that the vast majority of errors in a converged model are not due to an inherent inability to learn the sample, but rather a failure to maintain stable decision boundaries for previously "solved" cases. This "forgetting" appears to be a fundamental weakness in current DNN training paradigms across various architectures, including ShuffleNet and MobileNet.

Flipping Error Reduction (FER) Methodology

The FER method aims to preserve the "correct" knowledge a model acquires during training through a selective regularization process that anchors the model's behavior once it correctly identifies a sample.

Overview of the Flipping Error Reduction (FER) training process.

The core mechanism, as shown in Figure 3, involves three primary components:

  1. Weighted Average Behavior: The model maintains a running weighted average of its "soft" output distributions (class probabilities) for every sample it classifies correctly. Predictions from later epochs and those with higher confidence are assigned greater weights to form a stable reference behavior.
  2. Selective Regularization: A penalty (KL-Divergence) is applied only when the model currently predicts a sample correctly. This forces the current prediction to remain consistent with the historical "weighted average behavior," effectively anchoring the correct decision boundary.
  3. Freedom to Learn: Crucially, if the model misclassifies a sample in the current epoch, no regularization is applied. This provides the model with the necessary "free space" to explore the loss landscape and correct its mistakes without being tethered to a previous incorrect state.

Experimental Results

The effectiveness of FER was validated across a diverse range of benchmarks, including CIFAR-100, Tiny ImageNet, and fine-grained datasets like Stanford Dogs and CUB-200.

  • Generalization Performance: FER consistently outperformed standard training and traditional Label Smoothing. On the CUB-200 dataset, FER improved ResNet-18 accuracy by 6.56%.
  • Robustness to Label Noise: FER demonstrated significant resilience in noisy environments. In a scenario with 80% noisy labels, FER maintained an accuracy of 43.36%, whereas the standard baseline dropped to 12.37%. By relying on its own history of correct predictions, the model avoids being misled by corrupted labels.
  • Efficiency: Since the weighted average behavior is updated during the standard forward pass and regularization occurs only during training, FER introduces zero overhead in terms of model parameters or inference time.

Conclusion

Flipping Error Reduction (FER) addresses a fundamental yet overlooked weakness in DNN training: the instability of correct predictions over time. By selectively regularizing the model to maintain its correct historical behaviors, FER significantly enhances accuracy and robustness across diverse tasks. While it requires some additional memory to store soft behaviors during training, its zero-cost inference makes it a highly practical addition to the deep learning toolkit.

Summary updated

Abstract

Deep neural networks (DNNs) have been widely applied in various domains in artificial intelligence including computer vision and natural language processing. A DNN is typically trained for many epochs and then a validation dataset is used to select the DNN in an epoch (we simply call this epoch "the last epoch") as the final model for making predictions on unseen samples, while it usually cannot achieve a perfect accuracy on unseen samples. An interesting question is "how many test (unseen) samples that a DNN misclassifies in the last epoch were ever correctly classified by the DNN before the last epoch?". In this paper, we empirically study this question and find on several benchmark datasets that the vast majority of the misclassified samples in the last epoch were ever classified correctly before the last epoch, which means that the predictions for these samples were flipped from "correct" to "wrong". Motivated by this observation, we propose to restrict the behavior changes of a DNN on the correctly-classified samples so that the correct local boundaries can be maintained and the flipping error on unseen samples can be largely reduced. Extensive experiments on different benchmark datasets with different modern network architectures demonstrate that the proposed flipping error reduction (FER) approach can substantially improve the generalization, the robustness, and the transferability of DNNs without introducing any additional network parameters or inference cost, only with a negligible training overhead.
BibTeX
@inproceedings{deng2022reducing,
  author = {Xiang Deng and Yun Xiao and Bo Long and Zhongfei Zhang},
  title = {{Reducing Flipping Errors in Deep Neural Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i6.20603},
}
Figure