Reducing Flipping Errors in Deep Neural Networks

Xiang Deng, Yun Xiao, Bo Long, Zhongfei Zhang

Abstract

Deep neural networks (DNNs) are typically trained for many epochs, with the final model selected from the "last epoch" based on validation performance. However, these models often fail to achieve perfect accuracy on unseen samples. This paper investigates a critical question: how many test samples misclassified in the last epoch were actually classified correctly by the model at an earlier stage? Our empirical study reveals that the vast majority of final misclassifications are "flipping errors"—samples where the model's prediction shifted from correct to wrong. To address this, we propose Flipping Error Reduction (FER), a regularization approach that restricts behavior changes on correctly classified samples to maintain stable local boundaries. Extensive experiments demonstrate that FER substantially improves generalization, robustness, and transferability with negligible training overhead and zero additional inference cost.

The Flipping Error Phenomenon

In standard deep learning workflows, training is a dynamic process where decision boundaries shift as the model optimizes its loss function. The authors identify a frustrating phenomenon: Relative Flipping Error (RFE). By analyzing models on benchmarks like CIFAR-100, they found that over 90% of the samples a model gets wrong at the end of training were actually handled correctly at some earlier point.

This suggests that the final errors are not necessarily due to the model's inability to learn complex features, but rather a failure to retain what it has already mastered. As the model continues to learn new samples, it inadvertently pushes the decision boundaries of previously learned samples into the "wrong" zone.

Flipping Error Reduction (FER)

The proposed FER method introduces a simple yet effective mechanism to "anchor" the model’s behavior once it has successfully learned a sample. The core idea is to preserve the local decision boundaries that led to the correct classification.

1. Defining "Behavior" via Soft Distributions

Rather than just monitoring the final predicted class (hard label), FER focuses on the soft distribution (the probabilities assigned to all possible classes). This distribution captures the nuance of the local boundary—for instance, how much a specific image of a "husky" resembles a "wolf" or a "dog."

2. Weighted Historical Anchoring

For every sample in the training set, the model maintains a running weighted average of its soft distributions from all previous epochs where it achieved a correct classification. This historical average serves as a behavioral anchor. Higher weights are assigned to later epochs and predictions made with higher confidence.

3. Selective Regularization

During the training process, the loss function is modified to include a regularization term based on Kullback-Leibler (KL) Divergence:

  • If a sample was previously correct: The model is penalized if its current output deviates significantly from its historical "best" behavior. This prevents the model from "un-learning" the sample.
  • If a sample has never been correct: The model is left free to explore the feature space using standard cross-entropy loss.

Experimental Results

The researchers evaluated FER across a wide variety of architectures (e.g., ResNet-18) and tasks, ranging from general image classification to tabular data analysis.

Generalization and Accuracy

FER consistently outperformed standard training and common regularization techniques like Label Smoothing or Knowledge Distillation. Notable improvements included:

  • CIFAR-100: A ~3% accuracy boost for ResNet-18.
  • CUB-200-2011 (Fine-grained birds): A significant ~6.5% improvement in accuracy.
  • Tabular Data: Improved performance on datasets like Arcene and Iris, proving the method is not limited to computer vision.

Robustness to Noisy Labels

One of the most striking findings was FER's performance in high-noise environments. In experiments where 60% of training labels were intentionally corrupted (randomly flipped), FER maintained a last-epoch accuracy of 61.57%. In contrast, standard training collapsed to 32.22%. By anchoring to its own historical correct predictions, FER effectively ignores the influence of noisy labels once it finds the true underlying pattern.

Transferability

Models trained with FER produced more stable and fundamental features. When features learned on CIFAR-100 were applied to the STL-10 dataset (transfer learning), the FER-trained models showed superior performance, suggesting that reducing flipping errors leads to more robust feature representation.

Conclusion and Efficiency

Flipping Error Reduction (FER) offers a powerful way to improve DNN performance by ensuring that models do not forget what they have already learned. Because the "anchors" are only utilized during the training phase, the final deployed model has the exact same architecture and inference speed as a standard model. While it requires additional memory to store historical soft labels, the training overhead is negligible, making it a highly practical solution for improving the reliability of deep learning models.

Summary updated

Abstract

Deep neural networks (DNNs) have been widely applied in various domains in artificial intelligence including computer vision and natural language processing. A DNN is typically trained for many epochs and then a validation dataset is used to select the DNN in an epoch (we simply call this epoch "the last epoch") as the final model for making predictions on unseen samples, while it usually cannot achieve a perfect accuracy on unseen samples. An interesting question is "how many test (unseen) samples that a DNN misclassifies in the last epoch were ever correctly classified by the DNN before the last epoch?". In this paper, we empirically study this question and find on several benchmark datasets that the vast majority of the misclassified samples in the last epoch were ever classified correctly before the last epoch, which means that the predictions for these samples were flipped from "correct" to "wrong". Motivated by this observation, we propose to restrict the behavior changes of a DNN on the correctly-classified samples so that the correct local boundaries can be maintained and the flipping error on unseen samples can be largely reduced. Extensive experiments on different benchmark datasets with different modern network architectures demonstrate that the proposed flipping error reduction (FER) approach can substantially improve the generalization, the robustness, and the transferability of DNNs without introducing any additional network parameters or inference cost, only with a negligible training overhead.
BibTeX
@inproceedings{deng2022reducing,
  author = {Xiang Deng and Yun Xiao and Bo Long and Zhongfei Zhang},
  title = {{Reducing Flipping Errors in Deep Neural Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i6.20603},
}
Figure