Asynchronous Sharpness-Aware Minimization For Fast and Accurate Deep Learning

Junhyuk Jo, Jihyun Lim, Sunwoo LeearXiv 2025 · DOI · arXiv

Introduction

Achieving high generalization performance in deep learning often involves navigating complex loss landscapes to find "flat minima." Sharpness-Aware Minimization (SAM) has emerged as a powerful optimization technique for this purpose, consistently improving model robustness across various tasks. However, SAM's benefits come at a steep computational cost: it requires two gradient computations per iteration—one to identify the direction of sharpest ascent and another to perform the actual parameter update. This "SAM tax" effectively doubles training time compared to standard Stochastic Gradient Descent (SGD), limiting its adoption in large-scale production environments.

The AsyncSAM Framework

To address this bottleneck, we introduce Asynchronous-Parallel SAM (AsyncSAM). Our approach breaks the sequential data dependency inherent in the original SAM algorithm through two primary innovations:

1. Asynchronous Perturbation

Standard SAM calculates a perturbation $\epsilon$ based on the gradient of the current model state $w_t$. AsyncSAM instead utilizes a "stale" gradient from a previous iteration ($\tau=1$) to compute the perturbation. This allows the perturbation step to be decoupled from the current gradient calculation, enabling parallel execution.

2. System-Aware Batch Size Scaling

AsyncSAM leverages heterogeneous hardware by offloading the perturbation step (gradient ascent) to a secondary resource, such as a CPU, while the primary model update (gradient descent) continues on the GPU. To prevent the slower resource from becoming a bottleneck, we dynamically scale the batch size of the perturbation step ($b'$) relative to the main batch size ($b$) using the ratio of processing times: $b' = (T_{fast} / T_{slow}) \times b$. This ensures that the perturbation computation is entirely "hidden" within the time taken for the main model update.

Empirical Justification: Gradient Stability

The viability of using stale gradients rests on the observation that gradients remain remarkably stable across consecutive training steps. We measured the cosine similarity between gradients at adjacent iterations across multiple datasets and architectures.

Cosine similarity of gradients during CIFAR-10 training remains consistently high, justifying the use of stale information for perturbations.

As shown in the figures, the cosine similarity typically exceeds 0.8 and often approaches 1.0 as training progresses. This high degree of stability confirms that the local geometry of the loss landscape does not change drastically between steps, allowing AsyncSAM to approximate the "sharpness" direction effectively without real-time gradient information.

High gradient stability observed on CIFAR-100 further supports the robustness of the asynchronous approach across different task complexities.

Performance and Results

Our evaluations across image classification (CIFAR, Tiny-ImageNet), audio (Google Speech Commands), and fine-tuning tasks (Vision Transformers) demonstrate that AsyncSAM provides the best of both worlds:

  • Generalization: AsyncSAM matches or slightly exceeds the accuracy of the original SAM. For instance, on CIFAR-10, AsyncSAM achieved 92.60% accuracy, outperforming both SAM (92.53%) and SGD (91.96%).
  • Efficiency: In terms of wall-clock time and throughput, AsyncSAM is nearly indistinguishable from SGD. It eliminates the 2x overhead of SAM, making sharpness-aware optimization practically "free" in terms of computational time.
  • Landscape Flattening: Qualitative analysis via loss landscape visualizations confirms that AsyncSAM successfully converges to flatter minima, mirroring the behavior of the more expensive original algorithm.

Conclusion

AsyncSAM represents a significant step forward in making advanced optimization techniques accessible for large-scale deep learning. By exploiting gradient stability and heterogeneous hardware, it removes the primary barrier to adopting sharpness-aware minimization, providing superior generalization without the traditional temporal overhead. Future work will focus on relaxing theoretical assumptions and exploring further scaling via data-parallelism.

Summary updated

BibTeX
@misc{jo2025asynchronous,
  author = {Junhyuk Jo and Jihyun Lim and Sunwoo Lee},
  title = {{Asynchronous Sharpness-Aware Minimization For Fast and Accurate Deep Learning}},
  year = {2025},
  eprint = {2503.11147},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.11147},
}
Figure