LSAM: Asynchronous Distributed Training with Landscape-Smoothed Sharpness-Aware Minimization

Yunfei Teng, Sixin ZhangarXiv 2025 · DOI · arXiv

Introduction

Sharpness-Aware Minimization (SAM) has emerged as a powerful technique for improving neural network generalization by targeting "flat" minima in the loss landscape. However, SAM faces a significant "scalability wall" in distributed training environments. Standard data parallelism effectively increases the global batch size, which leads to a rapid deterioration in SAM’s effectiveness. LSAM (Landscape-Smoothed Sharpness-Aware Minimization) is proposed to overcome this trade-off, enabling efficient, large-scale asynchronous distributed training without sacrificing generalization.

Problem Statement: The Scalability Wall

In distributed settings, researchers typically rely on data parallelism to scale training. While this speeds up computation, it increases the effective batch size. SAM’s performance is known to degrade as the batch size grows, forcing a choice between slow, small-batch training or fast, large-batch training with poor generalization. LSAM aims to break this bottleneck by reframing optimization as a sampling problem rather than a simple data-splitting task.

Methodology: Sampling Parallelism

LSAM introduces a new paradigm called sampling parallelism. Instead of merely partitioning data across workers, LSAM treats the training process as sampling from a "Landscape-Smoothed" Boltzmann distribution ($\pi_{LSAM}$).

Dual-Loop Architecture

The algorithm employs a dual-loop structure consisting of an inner loop for Langevin Dynamics (MCMC) to explore the parameter space and an outer loop for a Nesterov-momentum-based optimization step.

Algorithm 1: The LSAM optimization procedure featuring asynchronous sampling and synchronous optimization steps.

Asynchronous Updates and Kernel Smoothing

Unlike standard synchronous data-parallel SAM, LSAM allows workers to generate samples independently. A global server aggregates these asynchronously to update a "global center." Furthermore, LSAM integrates SAM’s adversarial perturbations with Gaussian kernel smoothing. This mathematical unification allows the optimizer to simultaneously search for "flat" minima (via SAM) and "deep" minima (via Gaussian smoothing), targeting the "Goldilocks" zone of the loss landscape: regions that are both wide and deep. The paper also provides a formal proof that LSAM matches the convergence rates of classical Stochastic Gradient Descent (SGD).

Experimental Setup

The robustness of LSAM was evaluated across several standard computer vision benchmarks and architectures:

  • SVHN: CNN-5 and ResNet-18.
  • CIFAR-10: VGG-16, ResNet-20, and WideResNet (WRN-16×10).
  • CIFAR-100: ResNet-34, WRN-28×10, and WRN-40×10.

Key Results

LSAM consistently outperformed standard data-parallel SAM (DP-SAM) and other distributed optimizers such as EASGD and LSGD.

Test error (%) vs. training epochs for CNN-5 and ResNet-18 on the SVHN dataset, comparing LSAM against various baselines.

On the SVHN dataset with ResNet-18, LSAM achieved a test error of 3.04%, compared to 3.49% for DP-SAM. Across all CIFAR experiments, LSAM demonstrated faster convergence and lower final error rates. The asynchronous nature of the algorithm also improves hardware efficiency by mitigating the impact of "stragglers" (slow worker nodes) in real-world distributed environments, allowing the method to scale to more GPUs without the typical SAM performance drop-off.

Conclusion and Future Work

LSAM successfully decouples batch size from generalization performance by utilizing sampling parallelism. While it introduces some communication overhead—inheriting the extra forward-backward pass of SAM and requiring double the communication cost per iteration compared to standard SGD—it offers a scalable path for high-performance distributed training. Future research will explore LSAM's performance at extreme scales, such as hundreds of GPUs, and its applicability to Large Language Model (LLM) pre-training.

Summary updated

Abstract

While Sharpness-Aware Minimization (SAM) improves generalization in deep neural networks by minimizing both loss and sharpness, it suffers from inefficiency in distributed large-batch training. We present Landscape-Smoothed SAM (LSAM), a novel optimizer that preserves SAM's generalization advantages while offering superior efficiency. LSAM integrates SAM's adversarial steps with an asynchronous distributed sampling strategy, generating an asynchronous distributed sampling scheme, producing a smoothed sharpness-aware loss landscape for optimization. This design eliminates synchronization bottlenecks, accelerates large-batch convergence, and delivers higher final accuracy compared to data-parallel SAM.
BibTeX
@misc{teng2025lsam,
  author = {Yunfei Teng and Sixin Zhang},
  title = {{LSAM: Asynchronous Distributed Training with Landscape-Smoothed Sharpness-Aware Minimization}},
  year = {2025},
  eprint = {2509.03110},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.03110},
}
Figure