Finding Flat Minima Through Weight Space Averaging

296 concepts186 papers1 proposal

When training deep neural networks, we typically select the single "best" checkpoint—usually the one at the very end of the training run or the one that achieved the lowest validation loss. However, standard optimization algorithms like Stochastic Gradient Descent (SGD) or Adam are highly sensitive to the noise of the final few mini-batches. As a result, the final weight configuration often lands near the periphery of a wide valley or inside a sharp, brittle local minimum.

Weight space averaging fundamentally changes this approach. Instead of choosing a single point in the parameter space, it aggregates multiple weight configurations—either from different time steps of a single training run or from distinct optimization trajectories. Geometrically, this simple mathematical average acts as a powerful regularizer, locating a solution at the exact center of a flat loss region.

The Geometry of Flat Minima

To understand why weight averaging works, we must look at the geometry of the loss landscape. High-performing neural networks often reside in wide, flat valleys where the loss remains consistently low. Solutions located in the center of these flat regions generalize significantly better to unseen data than solutions in sharp, narrow pits.

During the final phases of training, an optimizer with a constant or fluctuating learning rate does not settle perfectly into the center of a valley. Instead, it "wanders" or bounces around the perimeter of the flat region due to gradient noise.

Standard Optimization:
[Iteration 1] ──> [Iteration 2] ──> [Iteration 3] ──> [Final Checkpoint (Edge)]

Weight Space Averaging:
[Iteration 1] ──┐
[Iteration 2] ──┼──> [Arithmetic Mean] ──> [Centered Solution (Flat Minimum)]
[Iteration 3] ──┘

As Ben Athiwaratkun et al. demonstrate in Improving Consistency-Based Semi-Supervised Learning with Weight Averaging, taking the arithmetic mean of these wandering points geometrically projects the final parameters directly into the center of the valley.

This centered solution is uniquely robust against distribution shifts. When a model encounters out-of-distribution data or a new downstream task, the underlying loss landscape shifts slightly. A model sitting at the edge of a valley or in a sharp minimum will suddenly experience a massive spike in loss. Conversely, a model positioned squarely in the center of a flat region remains safely inside the low-loss zone even after the landscape deforms.

Temporal Averaging: Trajectories and Cooldowns

The most common implementation of this idea is temporal averaging, which combines checkpoints from a single training run. This can be done post-hoc or dynamically during training.

  • Stochastic Weight Averaging (SWA): SWA samples weights at regular intervals during the late stages of training. Peng Lu et al. show in Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging that applying SWA to Transformer fine-tuning consistently locates flatter minima, smoothing out optimization noise across the final trajectory without requiring multiple gradient computations per step.
  • Exponential Moving Average (EMA): Rather than a simple average at the end, EMA maintains a running, exponentially decayed average of the weights throughout training. This approach is highly effective for stabilizing volatile optimization landscapes. For instance, Yasin Yazici et al. demonstrate in The Unusual Effectiveness of Averaging in GAN Training that EMA decouples the optimization process from final model selection, providing a non-invasive way to stabilize Generative Adversarial Networks.
  • Schedule-Free Optimization: Traditional training relies on rigid learning rate schedules (like cosine decay) to force the model to settle down at the end of training. In Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Training, Minhak Song explores how optimization algorithms can implicitly perform weight averaging continuously by interpolating between current iterates and a running average of past weights. This allows the model to maintain high-quality generalization performance at any point during training, bypassing the need for a dedicated cooling period.

Cross-Run Averaging and the Thin Shell Topology

Weight averaging is not restricted to a single chronological path. It can also be applied across entirely different training runs initialized from the same pre-trained checkpoint.

When multiple models are fine-tuned independently on different data subsets, hyperparameters, or random seeds, their weights diverge but remain within a shared low-loss basin. Averaging these independent models acts as a form of weight-space ensembling. Devansh Arpit et al. leverage this property in Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization, showing that averaging parameters across diverse runs cancels out the high variance and idiosyncratic noise of individual domains.

This multi-model behavior is governed by a distinct geometric structure. In Model Stock: All We Need Is Just a Few Fine-Tuned Models, Dong-Hwan Jang et al. reveal that fine-tuned weights tend to reside on a "thin shell" surrounding a central, optimal point.

       [Fine-Tuned Model A]
             \
              \
               v
       (Optimal Center) <─── [Fine-Tuned Model B]
               ^
              /
             /
       [Pre-trained Base]

By understanding this geometric topology, the authors demonstrate that you do not need a massive pool of candidate checkpoints to perform effective model merging. Instead, a precise mathematical interpolation of as few as two fine-tuned models can accurately approximate the optimal center of the weight space.

Why It Matters: The Free Lunch of Deep Learning

The primary advantage of weight space averaging is its computational efficiency at deployment.

Traditional ensembling requires running multiple models at inference time, multiplying the computational cost, memory footprint, and latency by the number of models in the ensemble. Weight space averaging provides a "free lunch": because the parameters are averaged before deployment, the resulting model has the exact same architecture, size, and inference speed as a standard single model.

By shifting the ensembling effect from the model outputs to the parameter space, weight averaging offers a zero-cost mechanism to achieve superior generalization, smoother optimization landscapes, and heightened robustness to real-world distribution shifts.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 186 papers
  2. Version 1 · 2026-10-04 · 275 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_stochastic_weight_averaging_37f3fa81,
  title = {stochastic weight averaging},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stochastic-weight-averaging/rdr_37f3fa81cb8c4d748008de4790e35e27},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_stochastic_weight_averaging_37f3fa81_v2,
  title = {stochastic weight averaging},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stochastic-weight-averaging/rdr_37f3fa81cb8c4d748008de4790e35e27/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake