iFlood: A Stable and Effective Regularizer

Yuexiang Xie, Zhen Wang, Yaliang Li, Ce Zhang, Jingren Zhou, Bolin DingICLR 2022 · OpenReview

TL;DR

iFlood is a simple yet powerful regularization technique that prevents overfitting by forcing the loss of each individual training sample to stay near a target "flood level." By shifting from global average constraints to instance-level ones, it resolves the instability issues of the original Flooding method and provides a robust framework for handling noisy or biased training data.

The Overfitting Dilemma and the Flaws of Flooding

Modern deep learning models are notoriously overparameterized, meaning they have enough capacity to perfectly memorize training data—including noise. This leads to the "generalization gap," where a model performs perfectly on training data but fails on new, unseen data.

A recent technique called Flooding attempted to solve this by preventing the averaged training loss from dropping to zero. It uses a "flood level" ($b$); if the average loss drops below $b$, the gradients are reversed to push the loss back up. However, further studies uncover two critical flaws in Flooding:

  1. Instability: Flooding only cares about the average. A model could achieve the target average loss by perfectly memorizing half the samples (zero loss) and completely failing on the other half (high loss). This leads to inconsistent performance.
  2. Implementation Discrepancy: In practice, models are trained using mini-batch SGD. The way Flooding is calculated over a batch creates a mathematical discrepancy between the intended objective and the actual optimization, leading to random, unstable training behavior.

Methods: Introducing Individual Flood (iFlood)

To resolve these issues, the authors propose individual Flood (iFlood). The core idea is deceptively simple: apply the flooding operation to every single instance rather than the batch average.

The iFlood loss function is defined as: $$\mathcal{L}_{\text{iFlood}} = \frac{1}{N} \sum_{i=1}^N (| \mathcal{L}_i - b | + b)$$

Where $\mathcal{L}_i$ is the loss of the $i$-th instance and $b$ is the hyperparameter "flood level."

Comparison of individual loss distributions between original Flooding and iFlood.

What makes iFlood fundamentally different? * Local Convergence: Instead of "global convergence" (where only the average loss $\approx b$), iFlood pursues "local convergence" (individual loss $\approx b$ for every sample). As seen in the distributions above, iFlood tightly clusters the losses around the target level, whereas original Flooding allows them to spread wildly. * Mathematical Consistency: Because the absolute value is inside the summation, the gradient calculated via mini-batch SGD is an unbiased estimator of the true gradient. This eliminates the randomness and discrepancy issues found in original Flooding. * Confidence Capping: It effectively prevents the model from becoming over-confident on any single sample, which is particularly useful when some samples might be mislabeled.

Key Insights and Theoretical Grounding

  • The "Aha" Moment on Stability: The authors show that Flooding's instability stems from its indifference to individual sample behavior. By forcing every sample to stay near the flood level, iFlood ensures the model treats the entire dataset consistently, leading to much lower variance in test results.
  • Denoising Theory: The paper provides a theoretical proof that iFlood acts as a surrogate for the maximum likelihood estimator when dealing with noisy labels. Essentially, $b$ acts as a "trust parameter"—it tells the model how much to believe the labels.
  • Less Inductive Bias: Compared to Label Smoothing (which assumes incorrect classes should have a uniform probability), iFlood is more flexible. It discounts the confidence of the predicted class but doesn't force the "leftover" probability into a specific distribution, allowing the model to learn more natural relationships between classes.

Key Results

The authors validated iFlood across an impressive array of benchmarks, including image classification (CIFAR-10/100, SVHN, ImageNet), language understanding (the full GLUE benchmark), and recommendation systems (the 45-million instance Criteo dataset).

Test accuracy comparison across various image and language datasets.
  • Superior Accuracy: As shown in the table above, iFlood consistently outperformed original Flooding, Label Smoothing, and Confident Penalty across diverse modalities. For example, on CIFAR-100, iFlood achieved 79.06% accuracy compared to Flooding's 78.63% and the unregularized 78.24%.
  • Robustness to Noise: In high-noise scenarios (e.g., 40% noisy labels injected into CIFAR-10 and SST-2), iFlood maintained significantly higher accuracy than all other methods, confirming its effectiveness as a denoiser.
  • Stability: The standard deviation of test accuracy across multiple runs was much lower for iFlood. On CIFAR-10, the variance was nearly 4x lower than that of original Flooding.
  • Large Scale Success: On the massive Criteo dataset, iFlood provided a 0.001-level improvement in AUC, which is considered highly significant in the advertising industry.

Limitations and Future Work

  • Hyperparameter Tuning: Like its predecessor, iFlood requires tuning the flood level $b$. While the paper suggests $b$ is related to the noise level or bias in the data, finding the optimal value still requires a grid search.
  • Computational Overhead: Calculating instance-level absolute values and their gradients can sometimes be slightly more computationally intensive in certain deep learning frameworks compared to batch-level operations, though the authors note this is generally negligible.
  • Future Directions: The authors suggest exploring adaptive flood levels that could change during training or vary dynamically for different types of samples.

Summary updated

Abstract

Various regularization methods have been designed to prevent overfitting of machine learning models. Among them, a surprisingly simple yet effective one, called Flooding, is proposed recently, which directly constrains the training loss on average to stay at a given level. However, our further studies uncover that the design of the loss function of Flooding can lead to a discrepancy between its objective and implementation, and cause the instability issue. To resolve these issues, in this paper, we propose a new regularizer, called individual Flood (denoted as iFlood). With instance-level constraints on training loss, iFlood encourages the trained models to better fit the under-fitted instances while suppressing the confidence on over-fitted ones. We theoretically show that the design of iFlood can be intrinsically connected with removing the noise or bias in training data, which makes it suitable for a variety of applications to improve the generalization performances of learned models. We also theoretically link iFlood to some other regularizers by comparing the inductive biases they introduce. Our experimental results on both image classification and language understanding tasks confirm that models learned with iFlood can stably converge to solutions with better generalization ability, and behave consistently at instance-level.
BibTeX
@inproceedings{xie2022iflood,
  author = {Yuexiang Xie and Zhen WANG and Yaliang Li and Ce Zhang and Jingren Zhou and Bolin Ding},
  title = {{iFlood: A Stable and Effective Regularizer}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=MsHnJPaBUZE},
}
Figure