Introduction
Standard training of deep neural networks (DNNs) is often assumed to be a progressive improvement toward an optimal state. However, empirical evidence suggests that models frequently "forget" correct predictions made during earlier stages of training. Researchers from SUNY Binghamton and JD.com identified this phenomenon as Flipping Errors—instances where a sample correctly classified in a previous epoch is misclassified in the final model state. To address this, they proposed Flipping Error Reduction (FER), a regularization technique that anchors a model's correct behaviors without increasing inference latency or parameter count.
The Flipping Error Phenomenon
The researchers introduced the Relative Flipping Error (RFE) metric to quantify the extent of this issue. RFE measures the proportion of final misclassifications that were correctly predicted at least once during the training process.
As illustrated in Figure 2, RFEs are remarkably high—often exceeding 90% for architectures like ResNet-18 on CIFAR-100. This suggests that the vast majority of errors in a converged model are not due to an inherent inability to learn the sample, but rather a failure to maintain stable decision boundaries for previously "solved" cases. This "forgetting" appears to be a fundamental weakness in current DNN training paradigms across various architectures, including ShuffleNet and MobileNet.
Flipping Error Reduction (FER) Methodology
The FER method aims to preserve the "correct" knowledge a model acquires during training through a selective regularization process that anchors the model's behavior once it correctly identifies a sample.
The core mechanism, as shown in Figure 3, involves three primary components:
- Weighted Average Behavior: The model maintains a running weighted average of its "soft" output distributions (class probabilities) for every sample it classifies correctly. Predictions from later epochs and those with higher confidence are assigned greater weights to form a stable reference behavior.
- Selective Regularization: A penalty (KL-Divergence) is applied only when the model currently predicts a sample correctly. This forces the current prediction to remain consistent with the historical "weighted average behavior," effectively anchoring the correct decision boundary.
- Freedom to Learn: Crucially, if the model misclassifies a sample in the current epoch, no regularization is applied. This provides the model with the necessary "free space" to explore the loss landscape and correct its mistakes without being tethered to a previous incorrect state.
Experimental Results
The effectiveness of FER was validated across a diverse range of benchmarks, including CIFAR-100, Tiny ImageNet, and fine-grained datasets like Stanford Dogs and CUB-200.
- Generalization Performance: FER consistently outperformed standard training and traditional Label Smoothing. On the CUB-200 dataset, FER improved ResNet-18 accuracy by 6.56%.
- Robustness to Label Noise: FER demonstrated significant resilience in noisy environments. In a scenario with 80% noisy labels, FER maintained an accuracy of 43.36%, whereas the standard baseline dropped to 12.37%. By relying on its own history of correct predictions, the model avoids being misled by corrupted labels.
- Efficiency: Since the weighted average behavior is updated during the standard forward pass and regularization occurs only during training, FER introduces zero overhead in terms of model parameters or inference time.
Conclusion
Flipping Error Reduction (FER) addresses a fundamental yet overlooked weakness in DNN training: the instability of correct predictions over time. By selectively regularizing the model to maintain its correct historical behaviors, FER significantly enhances accuracy and robustness across diverse tasks. While it requires some additional memory to store soft behaviors during training, its zero-cost inference makes it a highly practical addition to the deep learning toolkit.
Summary updated