Abstract
Sharpness-Aware Minimization (SAM) has emerged as a powerful technique to enhance model generalization by seeking flatter regions in the loss landscape. However, its computational cost—requiring two forward and backward passes per update—doubles the training time compared to standard Stochastic Gradient Descent (SGD). This paper introduces Asymptotic Unbiased SAM (AUSAM), a method that reduces this overhead by approximately 70%. By utilizing a "loss-difference" proxy to dynamically sample the most informative data points in each batch, AUSAM achieves the speed of standard SGD while maintaining or even exceeding the performance gains of vanilla SAM.
Introduction
Generalization remains a central challenge in deep learning. Recent research indicates that models converging to "flat" minima in the loss landscape tend to generalize better than those in "sharp" peaks. Sharpness-Aware Minimization (SAM) explicitly targets these flat regions by perturbing model weights during training. Despite its effectiveness, the primary drawback of SAM is its computational inefficiency; it is exactly twice as slow as SGD. Existing "Fast SAM" variants often compromise accuracy or introduce significant complexity. AUSAM addresses this by keeping SAM’s flatness-seeking capabilities while significantly improving training efficiency through intelligent sample selection.
Methodology: Dynamic Sample Selection
The core innovation of AUSAM is its approach to dynamic sample selection. Rather than applying the expensive SAM update to every sample in a mini-batch, AUSAM focuses on a subset of "important" samples.
- Importance Criterion: Theoretically, samples with larger Gradient Norms (GNS) are most critical for preserving generalization.
- The "Free" Proxy (DLP): Since computing individual gradient norms is slow, the authors use the absolute difference in loss before and after SAM perturbation ($|L(w + \epsilon) - L(w)|$), termed Difference in Loss before and after Perturbation (DLP). This score is essentially free as SAM already computes these losses.
- Probabilistic Sampling: To prevent overfitting, a moving average of DLP scores (ADLP) is used to probabilistically sample a subset of the batch, controlled by a hyperparameter $\alpha$.
- Asymptotic Unbiasedness: The method ensures that as the model converges, the bias between the sampled subset and the full batch diminishes, maintaining training stability.
Experimental Evaluation
The authors validated AUSAM across diverse benchmarks, including image classification (CIFAR-10/100, Tiny-ImageNet), Human Pose Estimation (MPII), and Quantization-Aware Training. Architectures tested range from ResNet-18/50 and WideResNet-28-10 to MobileNetv1 and PyramidNet-110.
Key Results
AUSAM demonstrates a significant performance-efficiency trade-off across all tested scenarios.
- Efficiency Gains: With a sampling rate of $\alpha=0.5$, AUSAM achieves a 70% reduction in SAM's overhead. In many cases, it matches the wall-clock time of standard SGD.
- Accuracy Preservation: On CIFAR-100 with WideResNet-28-10, AUSAM-0.6 reached 84.73% accuracy, slightly surpassing vanilla SAM (84.71%) with much higher efficiency.
- Scalability: The method scales effectively to larger datasets like Tiny-ImageNet, where it continues to provide substantial speedups without significant accuracy loss.
Discussion and Insights
AUSAM's success stems from using the SAM mechanism itself as a diagnostic tool. By identifying high-gradient samples—which provide a high-fidelity signal of the loss landscape's sharpness—the optimizer can focus its computational budget where it matters most. Furthermore, being architecture-agnostic, AUSAM serves as a "plug-and-play" solution for various tasks, including specialized ones like pose estimation and model quantization.
Limitations
While highly effective, AUSAM requires tuning the sampling ratio ($\alpha$) and normalization bounds. Smaller networks like MobileNet show higher sensitivity to sample omission, requiring less aggressive importance scoring. Additionally, the method requires a brief "warm-up" period to stabilize importance scores before speed gains are fully realized.
Conclusion
AUSAM offers a compelling solution to the computational bottleneck of Sharpness-Aware Minimization. By intelligently sampling informative data points using a zero-cost proxy, it bridges the gap between the generalization benefits of SAM and the efficiency of SGD.
Summary updated