When training deep neural networks, standard optimizers like Stochastic Gradient Descent (SGD) or Adam focus entirely on finding a single point in the parameter space where the training loss is minimized. This approach, known as Empirical Risk Minimization (ERM), often leads the model into "sharp" minima—narrow, steep ravines in the loss landscape. While these points show low loss on the training data, a tiny shift in the data distribution at test time can push the model up the steep walls of the ravine, causing a massive spike in error.
To solve this, researchers have shifted toward finding "flat" minima: broad, stable valleys where the loss remains consistently low even if the weights are slightly perturbed. The pioneering framework for this is Sharpness-Aware Minimization (SAM), introduced by Pierre Foret et al. in Sharpness-Aware Minimization for Efficiently Improving Generalization. SAM explicitly minimizes both the training loss and the landscape sharpness, yielding substantial gains in generalization, label-noise robustness, and model compressibility.
However, these gains come with a severe catch: standard sharpness-aware optimization effectively doubles the computational cost of training, creating a major bottleneck for modern deep learning workflows.
The Mechanics of the $2\times$ Computational Penalty
To understand why sharpness-aware optimization is so expensive, we have to look at how it uncovers flat regions. Instead of calculating a gradient at the current weight position $w$, SAM solves a local minimax problem: it searches for the worst-case adversarial perturbation $\epsilon$ within a small neighborhood $\rho$, and then updates the weights to minimize the loss at that worst-case location.
$$\min_w \max_{\|\epsilon\| \le \rho} L(w + \epsilon)$$
In practice, this mathematical formulation forces the optimizer to perform a two-step dance during every single parameter update:
- The Ascent Step (Inner Maximization): The model performs a standard forward pass and a backward pass to compute the gradient at the current weights $w$. This gradient points in the direction of the steepest ascent. The optimizer uses this information to calculate the adversarial perturbation $\epsilon$, temporarily shifting the weights to $w + \epsilon$.
- The Descent Step (Outer Minimization): At this new perturbed position, the model must perform a second forward pass and a second backward pass to compute the gradient of the perturbed loss. This final gradient is what actually updates the original weights $w$.
As detailed by Jungmin Kwon et al. in ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks and Boqing Gong et al. in Surrogate Gap Minimization Improves Sharpness-Aware Training, this dual-pass requirement means every single training step demands two full forward-backward cycles. The wall-clock training time and the number of floating-point operations (FLOPs) instantly double compared to standard first-order optimizers.
Where the Overhead Hurts Most
This 100% computational penalty is not just an inconvenience; it represents a critical deployment barrier across several domains:
- Federated Learning and Edge Devices: In distributed environments, local edge devices (like smartphones or IoT sensors) train models on decentralized data. While seeking flat minima helps prevent models from drifting and overfitting to local data, Barbara Caputo et al. (Improving Generalization in Federated Learning by Seeking Flat Minima) and Zhe Qu et al. (Generalized Federated Learning via Sharpness Aware Minimization) emphasize that doubling the local computational burden severely strains the strict battery life and processing constraints of edge hardware.
- Large-Scale Production Models: For massive architectures like Large Language Models (LLMs) or industrial Click-Through Rate (CTR) prediction networks, training budgets are already pushed to absolute hardware limits. As noted in work on language models (Sharpness-Aware Minimization Improves Language Model Generalization), even a fractional increase in overhead can make an optimization strategy economically or physically unfeasible.
Bypassing the Dual-Pass Penalty
To preserve the generalization benefits of flat minima without paying the steep dual-pass tax, a major research direction focuses on developing algorithms that approximate or achieve landscape flatness with minimal overhead.
Trajectory-Based Estimation
Instead of generating an explicit adversarial perturbation at every step via an extra gradient calculation, why not use the information the optimizer already collects? Jiawei Du proposed this exact intuition in Sharpness-Aware Training for Free. By tracking the model's own weight update trajectory over recent steps, the Sharpness-Aware Training for Free (SAF) algorithm estimates the local landscape roughness. This allows the model to avoid the inner maximization step entirely, achieving SAM-level generalization with virtually zero additional computational overhead.
Amortization and Delayed Activation
Another intuitive strategy is to realize that a model does not need to search for flat minima at every single second of its training run. Maksym Andriushchenko et al. demonstrated in Towards Understanding Sharpness-Aware Minimization that the implicit bias of sharpness-aware training can often be captured by applying the dual-pass mechanism selectively—such as only during the final stages of training or on specific parameter subsets—dramatically cutting down the cumulative training time.
Alternative Flatness Objectives
Rather than relying on the expensive minimax formulation to find flat regions, some approaches swap out the adversarial search for cheaper statistical alternatives. For instance, in parameter-efficient fine-tuning, Tao Li et al. developed Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape, which utilizes an efficient Bayesian expectation loss instead of adversarial optimization, bypassing the dual-pass bottleneck entirely.
Manipulating Stochastic Gradient Noise
Instead of exploring a neighborhood explicitly, some methods alter the inherent "noise" of the optimization process to naturally guide the model toward flat regions. Zeke Xie et al. introduced this concept in Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalization. By maintaining two independent momentum terms and using their difference to alter stochastic gradient noise, the optimizer escapes sharp minima naturally without requiring any extra forward or backward passes.
Post-Hoc Geometric Averaging
Finally, researchers have found success by decoupling the pursuit of flat minima from the inner loop of the optimizer entirely. Peng Lu et al. demonstrated in Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging that simply applying Stochastic Weight Averaging (SWA) during the final epochs of training naturally shifts the model toward the center of flat loss valleys. Because weight averaging happens post-hoc or at long intervals, it provides the benefits of a smoothed loss landscape with zero per-step computational penalty.