Asymptotic Unbiased Sample Sampling to Speed Up Sharpness-Aware Minimization

Jiaxin Deng, Junbiao Pang, Baochang Zhang, Guodong GuoAAAI 2025 · DOI

Abstract

Sharpness-Aware Minimization (SAM) has emerged as a powerful technique to enhance model generalization by seeking flatter regions in the loss landscape. However, its computational cost—requiring two forward and backward passes per update—doubles the training time compared to standard Stochastic Gradient Descent (SGD). This paper introduces Asymptotic Unbiased SAM (AUSAM), a method that reduces this overhead by approximately 70%. By utilizing a "loss-difference" proxy to dynamically sample the most informative data points in each batch, AUSAM achieves the speed of standard SGD while maintaining or even exceeding the performance gains of vanilla SAM.

Introduction

Generalization remains a central challenge in deep learning. Recent research indicates that models converging to "flat" minima in the loss landscape tend to generalize better than those in "sharp" peaks. Sharpness-Aware Minimization (SAM) explicitly targets these flat regions by perturbing model weights during training. Despite its effectiveness, the primary drawback of SAM is its computational inefficiency; it is exactly twice as slow as SGD. Existing "Fast SAM" variants often compromise accuracy or introduce significant complexity. AUSAM addresses this by keeping SAM’s flatness-seeking capabilities while significantly improving training efficiency through intelligent sample selection.

Methodology: Dynamic Sample Selection

The core innovation of AUSAM is its approach to dynamic sample selection. Rather than applying the expensive SAM update to every sample in a mini-batch, AUSAM focuses on a subset of "important" samples.

  1. Importance Criterion: Theoretically, samples with larger Gradient Norms (GNS) are most critical for preserving generalization.
  2. The "Free" Proxy (DLP): Since computing individual gradient norms is slow, the authors use the absolute difference in loss before and after SAM perturbation ($|L(w + \epsilon) - L(w)|$), termed Difference in Loss before and after Perturbation (DLP). This score is essentially free as SAM already computes these losses.
  3. Probabilistic Sampling: To prevent overfitting, a moving average of DLP scores (ADLP) is used to probabilistically sample a subset of the batch, controlled by a hyperparameter $\alpha$.
  4. Asymptotic Unbiasedness: The method ensures that as the model converges, the bias between the sampled subset and the full batch diminishes, maintaining training stability.

Experimental Evaluation

The authors validated AUSAM across diverse benchmarks, including image classification (CIFAR-10/100, Tiny-ImageNet), Human Pose Estimation (MPII), and Quantization-Aware Training. Architectures tested range from ResNet-18/50 and WideResNet-28-10 to MobileNetv1 and PyramidNet-110.

Key Results

AUSAM demonstrates a significant performance-efficiency trade-off across all tested scenarios.

Comparison of AUSAM with state-of-the-art methods on CIFAR-10 and CIFAR-100 datasets. AUSAM-0.6 often matches or exceeds SAM's accuracy while being significantly faster (higher SAM% indicates higher speed).
  • Efficiency Gains: With a sampling rate of $\alpha=0.5$, AUSAM achieves a 70% reduction in SAM's overhead. In many cases, it matches the wall-clock time of standard SGD.
  • Accuracy Preservation: On CIFAR-100 with WideResNet-28-10, AUSAM-0.6 reached 84.73% accuracy, slightly surpassing vanilla SAM (84.71%) with much higher efficiency.
  • Scalability: The method scales effectively to larger datasets like Tiny-ImageNet, where it continues to provide substantial speedups without significant accuracy loss.
Performance of AUSAM on the Tiny-ImageNet dataset, showing competitive accuracy with significant speedups across different ResNet and MobileNet architectures.

Discussion and Insights

AUSAM's success stems from using the SAM mechanism itself as a diagnostic tool. By identifying high-gradient samples—which provide a high-fidelity signal of the loss landscape's sharpness—the optimizer can focus its computational budget where it matters most. Furthermore, being architecture-agnostic, AUSAM serves as a "plug-and-play" solution for various tasks, including specialized ones like pose estimation and model quantization.

Limitations

While highly effective, AUSAM requires tuning the sampling ratio ($\alpha$) and normalization bounds. Smaller networks like MobileNet show higher sensitivity to sample omission, requiring less aggressive importance scoring. Additionally, the method requires a brief "warm-up" period to stabilize importance scores before speed gains are fully realized.

Conclusion

AUSAM offers a compelling solution to the computational bottleneck of Sharpness-Aware Minimization. By intelligently sampling informative data points using a zero-cost proxy, it bridges the gap between the generalization benefits of SAM and the efficiency of SGD.

Summary updated

BibTeX
@inproceedings{deng2025asymptotic,
  author = {Jiaxin Deng and Junbiao Pang and Baochang Zhang and Guodong Guo},
  title = {{Asymptotic Unbiased Sample Sampling to Speed Up Sharpness-Aware Minimization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i15.33780},
}
Figure