Abstract
Deep neural networks (DNNs) are typically trained for many epochs, with the final model selected from the "last epoch" based on validation performance. However, these models often fail to achieve perfect accuracy on unseen samples. This paper investigates a critical question: how many test samples misclassified in the last epoch were actually classified correctly by the model at an earlier stage? Our empirical study reveals that the vast majority of final misclassifications are "flipping errors"—samples where the model's prediction shifted from correct to wrong. To address this, we propose Flipping Error Reduction (FER), a regularization approach that restricts behavior changes on correctly classified samples to maintain stable local boundaries. Extensive experiments demonstrate that FER substantially improves generalization, robustness, and transferability with negligible training overhead and zero additional inference cost.
The Flipping Error Phenomenon
In standard deep learning workflows, training is a dynamic process where decision boundaries shift as the model optimizes its loss function. The authors identify a frustrating phenomenon: Relative Flipping Error (RFE). By analyzing models on benchmarks like CIFAR-100, they found that over 90% of the samples a model gets wrong at the end of training were actually handled correctly at some earlier point.
This suggests that the final errors are not necessarily due to the model's inability to learn complex features, but rather a failure to retain what it has already mastered. As the model continues to learn new samples, it inadvertently pushes the decision boundaries of previously learned samples into the "wrong" zone.
Flipping Error Reduction (FER)
The proposed FER method introduces a simple yet effective mechanism to "anchor" the model’s behavior once it has successfully learned a sample. The core idea is to preserve the local decision boundaries that led to the correct classification.
1. Defining "Behavior" via Soft Distributions
Rather than just monitoring the final predicted class (hard label), FER focuses on the soft distribution (the probabilities assigned to all possible classes). This distribution captures the nuance of the local boundary—for instance, how much a specific image of a "husky" resembles a "wolf" or a "dog."
2. Weighted Historical Anchoring
For every sample in the training set, the model maintains a running weighted average of its soft distributions from all previous epochs where it achieved a correct classification. This historical average serves as a behavioral anchor. Higher weights are assigned to later epochs and predictions made with higher confidence.
3. Selective Regularization
During the training process, the loss function is modified to include a regularization term based on Kullback-Leibler (KL) Divergence:
- If a sample was previously correct: The model is penalized if its current output deviates significantly from its historical "best" behavior. This prevents the model from "un-learning" the sample.
- If a sample has never been correct: The model is left free to explore the feature space using standard cross-entropy loss.
Experimental Results
The researchers evaluated FER across a wide variety of architectures (e.g., ResNet-18) and tasks, ranging from general image classification to tabular data analysis.
Generalization and Accuracy
FER consistently outperformed standard training and common regularization techniques like Label Smoothing or Knowledge Distillation. Notable improvements included:
- CIFAR-100: A ~3% accuracy boost for ResNet-18.
- CUB-200-2011 (Fine-grained birds): A significant ~6.5% improvement in accuracy.
- Tabular Data: Improved performance on datasets like Arcene and Iris, proving the method is not limited to computer vision.
Robustness to Noisy Labels
One of the most striking findings was FER's performance in high-noise environments. In experiments where 60% of training labels were intentionally corrupted (randomly flipped), FER maintained a last-epoch accuracy of 61.57%. In contrast, standard training collapsed to 32.22%. By anchoring to its own historical correct predictions, FER effectively ignores the influence of noisy labels once it finds the true underlying pattern.
Transferability
Models trained with FER produced more stable and fundamental features. When features learned on CIFAR-100 were applied to the STL-10 dataset (transfer learning), the FER-trained models showed superior performance, suggesting that reducing flipping errors leads to more robust feature representation.
Conclusion and Efficiency
Flipping Error Reduction (FER) offers a powerful way to improve DNN performance by ensuring that models do not forget what they have already learned. Because the "anchors" are only utilized during the training phase, the final deployed model has the exact same architecture and inference speed as a standard model. While it requires additional memory to store historical soft labels, the training overhead is negligible, making it a highly practical solution for improving the reliability of deep learning models.
Summary updated