Abstract
As models grow larger and more complex, achieving better off-sample generalization with minimal trial-and-error is critical to the reliability and economy of machine learning workflows. As a proxy for the well-studied heuristic of seeking "flat" local minima, gradient regularization is a natural avenue, and first-order approximations such as Flooding and sharpness-aware minimization (SAM) have received significant attention, but their performance depends critically on hyperparameters (flood threshold and neighborhood radius, respectively) that are non-trivial to specify in advance. In order to develop a procedure which is more resilient to misspecified hyperparameters, with the hard-threshold "ascent-descent" switching device used in Flooding as motivation, we propose a softened, pointwise mechanism called SoftAD that downweights points on the borderline, limits the effects of outliers, and retains the ascent-descent effect. We contrast formal stationarity guarantees with those for Flooding, and empirically demonstrate how SoftAD can realize classification accuracy competitive with SAM and Flooding while maintaining a much smaller loss generalization gap and model norm. Our empirical tests range from simple binary classification on the plane to image classification using neural networks with millions of parameters; the key trends are observed across all datasets and models studied, and suggest a potential new approach to implicit regularization.
Introduction
In modern deep learning, over-parameterized models often achieve zero training loss but converge to "sharp" minima that generalize poorly. To address this, "flooding" was introduced as a technique to prevent the training loss from reaching zero by forcing gradient ascent when the average loss drops below a threshold ($\theta$). However, flooding suffers from instability due to its "hard" switching mechanism, ignores individual data point performance by focusing on batch averages, and does not explicitly encourage low model complexity.
The SoftAD Mechanism
SoftAD (Soft Ascent-Descent) addresses these limitations by moving from a batch-level hard threshold to a pointwise soft wrapper. Instead of a binary switch, it utilizes a smooth, continuous pseudo-Huber function ($\rho$) to determine updates.
If an individual point's loss is significantly higher than $\theta$, it receives a standard descent update. If it is significantly lower, it receives an ascent update. Crucially, points near the threshold are downweighted, leading to Lipschitz continuous gradients. This smoothness allows for formal stationarity guarantees that are difficult to establish for the original flooding method.
Experimental Evaluation
The authors evaluated SoftAD against Empirical Risk Minimization (ERM), Flooding, and Sharpness-Aware Minimization (SAM) across several benchmarks:
- Synthetic 2D Data: Two Gaussians, Sinusoid, and Spiral datasets.
- Image Classification: CIFAR-10/100 (ResNet-34), FashionMNIST (Feedforward NN), and SVHN (ResNet-18).
Key Insights and Stability
SoftAD encourages the entire distribution of losses to concentrate near $\theta$, rather than just the mean. This stronger requirement acts as a powerful implicit regularizer. Furthermore, SoftAD is computationally efficient, requiring only one gradient computation per update—half the cost of SAM.
As shown in the optimization trajectories, SoftAD eliminates the oscillations characteristic of flooding, leading to more stable convergence. This stability is a key factor in its ability to maintain a smaller generalization gap.
Results
- Generalization Gap: SoftAD consistently achieved the smallest gap between training and test loss. On CIFAR-100, SoftAD's gap was 2.701, compared to 7.603 for ERM and 3.672 for SAM.
- Model Norms: SoftAD naturally keeps model weights small without explicit $L_2$ regularization. On CIFAR-10, SoftAD's model norm remained around 180, while ERM and SAM climbed to 220 and 240, respectively.
- Accuracy: SoftAD matched or nearly matched the classification accuracy of SAM and Flooding while maintaining lower test loss and smaller weights.
Conclusion and Future Work
SoftAD provides a stable, efficient alternative to existing gradient regularization techniques. While it still requires the manual setting of the threshold $\theta$, its pointwise soft-weighting mechanism offers significant advantages in generalization and model health. Future research will explore automated threshold selection based on Bayes error estimates and the behavior of SoftAD with non-cross-entropy loss functions.
Summary updated