Weight Averaging for Flatter Minima and Better Generalization

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2275 papers348 conceptsfrozen 2026-10-04

In the pursuit of high-performing neural networks, the standard approach is to select the single "best" set of weights found during training—usually the checkpoint with the lowest validation loss. However, a growing body of research suggests that this "winner-take-all" strategy is sub-optimal. Instead, by averaging multiple sets of weights from the same training trajectory or from different models, we can find solutions that generalize significantly better to unseen data.

The core intuition behind this direction is geometric: the loss landscape of a deep neural network is vast and rugged, but not all minima are created equal. Weight averaging acts as a mechanism to pull the model away from "sharp" minima and toward the center of "flat" regions.

The Geometry of Generalization

Standard Stochastic Gradient Descent (SGD) typically converges to the boundary of a wide, low-loss basin. Because the learning rate is usually decayed toward the end of training, the optimizer settles into a local minimum that may be sharp—meaning a small shift in the input data or a slight perturbation in weights could lead to a massive spike in loss.

Wesley J. Maddox, Pavel Izmailov, and colleagues introduced Stochastic Weight Averaging (SWA) in A Simple Baseline for Bayesian Uncertainty in Deep Learning to solve this. By maintaining a running average of weights during the final stages of training—often using a constant, higher learning rate—the model "traverses" the flat region of the loss landscape. Averaging these points effectively finds the centroid of the basin. As Weishi Li and others note in Deep Model Fusion: A Survey, these centralized regions are inherently more robust to distribution shifts.

Temporal vs. Parallel Averaging

Research into weight averaging generally follows two paths: vertical (temporal) and horizontal (parallel).

  1. Temporal Averaging (Vertical): This involves averaging checkpoints from a single training run. Fabian David Schmidt and Goran Glavaš demonstrate in Free Lunch: Robust Cross-Lingual Transfer via Model Checkpoint Averaging that this is a "free lunch"—it improves robustness in cross-lingual tasks without any extra inference cost. Similarly, Antti Tarvainen showed in Mean teachers are better role models that an exponential moving average (EMA) of weights creates a more stable "teacher" for semi-supervised learning than the raw weights themselves.
  2. Parallel Averaging (Horizontal): This involves training multiple models (often from the same initialization) and merging them. Moonseok Choi and Juho Lee explore this in Sparse Weight Averaging with Multiple Particles for Iterative Magnitude Pruning, where they use multiple "particles" to find a shared loss basin, leading to more robust sparse networks.

Xiaozhe Gu and colleagues unified these in Hierarchical Weight Averaging for Deep Neural Networks, showing that combining both temporal and parallel movement identifies even flatter regions than either method alone.

Why It Works: The Flatness Hypothesis

The recurring theme across these papers is that flatness equals generalization. In Robustness of Brain Tumor Segmentation, Sabine Müller and Norbert Graf argue that SWA helps models ignore high-frequency noise and focus on global structures. This is because sharp minima are often artifacts of specific noise in the training batch, while flat minima represent features that are consistent across the entire data distribution.

This principle holds even for massive Transformer models. Peng Lu and Ivan Kobyzev found in Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging that SWA consistently outperforms complex knowledge distillation methods. By locating the center of a wide basin, the model becomes less sensitive to the specific hyperparameters or noise encountered during fine-tuning.

Beyond Accuracy: Robustness and Fairness

Weight averaging is increasingly used as a tool for more than just Top-1 accuracy. In Seasoning Model Soups for Robustness to Adversarial and Natural Distribution Shifts, Francesco Croce and Evan Shelhamer show that "soups"—linear interpolations of weights—can trade off different types of robustness, allowing a model to handle both adversarial attacks and natural corruptions (like weather) simultaneously.

Furthermore, Yongshuo Zong and Timothy Hospedales suggest in MEDFAIR: Benchmarking Fairness for Medical Imaging that the way we select or average checkpoints is a primary driver of fairness. By seeking flatter minima, models learn features that are more consistent across different demographic groups (treated as different "domains"), rather than over-fitting to the majority group's specific features.

Conclusion

The shift toward weight-space averaging represents a move from "finding a solution" to "finding a stable region." Whether it is called SWA, Checkpoint Averaging, or Model Soups, the underlying mechanism remains the same: leveraging the diversity of the optimization trajectory to find a central, flat point in the loss landscape. This approach provides a rare "free lunch" in deep learning: improved generalization, stability, and robustness with zero additional cost at inference time.

Papers in this version

Showing 60 of 275 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 186 papers
  2. Version 1 · 2026-10-04 · 275 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_stochastic_weight_averaging_37f3fa81,
  title = {stochastic weight averaging},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stochastic-weight-averaging/rdr_37f3fa81cb8c4d748008de4790e35e27},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_weight_averaging_for_flatter_minima_and__37f3fa81_v1,
  title = {Weight Averaging for Flatter Minima and Better Generalization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/weight-averaging-for-flatter-minima-and-better-generalization/rdr_37f3fa81cb8c4d748008de4790e35e27/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}