SWAN: SGD with Normalization and Whitening Enables Stateless LLM Training

Chao Ma, Wenbo Gong, Meyer Scetbon, Edward Meeds

Abstract

Training Large Language Models (LLMs) is constrained by the significant memory overhead of optimizer states, often referred to as the "Adam Tax." We introduce SWAN (SGD with Whitening And Normalization), a stateless optimizer that eliminates the need for storing historical gradient moments. By pre-processing gradients with row-wise normalization and matrix whitening, SWAN achieves parity with Adam's performance while reducing total training memory by approximately 50%. Empirical evaluations on LLaMA models up to 1.3B parameters demonstrate that SWAN provides a >2x speedup in token efficiency without compromising throughput.

Introduction

The memory requirements for training state-of-the-art LLMs are immense. Beyond model weights, the first and second moments stored by optimizers like Adam can double the total memory footprint. While Stochastic Gradient Descent (SGD) is memory-optimal due to its stateless nature, it typically fails to converge effectively on the ill-conditioned loss landscapes of Transformers. SWAN bridges this gap by demonstrating that the internal structure of a single gradient matrix contains sufficient information to emulate adaptive optimization, allowing for efficient, stateless training.

Methodology

SWAN replaces history-tracking with two key gradient pre-processing operations:

  1. GradNorm: Row-wise normalization that stabilizes gradient distributions, acting as a gradient-space equivalent to RMSNorm.
  2. GradWhitening: A second-order update that "whitens" the gradient matrix using the operation $(GG^\top)^{-1/2}G$. This flattens the local geometry of the loss landscape, ensuring consistent progress across all directions.

To maintain computational feasibility, the authors developed NSDS (Newton-Schulz with Diagonal Substitution). This acceleration scheme reduces the complexity of matrix whitening to $O(m^2)$, ensuring that SWAN's per-step overhead is comparable to Adam's element-wise updates.

Experimental Setup

The researchers evaluated SWAN on LLM pre-training using the C4 (Colossal Clean Crawled Corpus). Experiments were conducted on LLaMA-style architectures at scales of 60M, 130M, 350M, and 1.3B parameters. Robustness was further verified using synthetic non-convex benchmarks like the Rastrigin function.

Results

Token Efficiency and Performance

SWAN consistently outperforms Adam in data efficiency. On a 1.3B parameter LLaMA model, SWAN reached the target evaluation perplexity using less than half the tokens required by Adam, representing a speedup of over 2.4x.

Test perplexity comparison on a 1.3B Llama model showing >2.4x speedup for SWAN over Adam.

Memory Footprint and Throughput

By eliminating optimizer states, SWAN achieves a ~50% reduction in end-to-end memory consumption. As shown in the comparison across model sizes, SWAN significantly lowers the memory barrier for training large models compared to Adam and even other memory-efficient methods like GaLore.

End-to-end memory consumption across different model sizes (1b, 7b, 13b) comparing Adam, GaLore, SWAN, and other baselines.

Importantly, SWAN maintains a training throughput nearly identical to Adam. The NSDS acceleration ensures that the benefits of whitening do not come at the cost of significantly slower training steps.

Discussion

A major insight of this work is that "structure over history" is a viable path for LLM optimization. SWAN's ability to neutralize curvature through whitening allows it to navigate narrow valleys in the loss landscape more effectively than standard SGD. Notably, SWAN's performance advantage appears to scale with model size, and it exhibits high robustness to learning rate schedules, often requiring no warm-up period.

Conclusion

SWAN presents a paradigm shift in LLM training by enabling high-performance, stateless optimization. By halving memory requirements and doubling token efficiency, it significantly lowers the resource threshold for training massive models. Future research will focus on validating these results at the 100B+ parameter scale and exploring SWAN's applicability to other deep learning domains.

Summary updated

BibTeX
@inproceedings{ma2025swan,
  author = {Chao Ma and Wenbo Gong and Meyer Scetbon and Edward Meeds},
  title = {{SWAN: SGD with Normalization and Whitening Enables Stateless LLM Training}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=CQZXGmw5vO},
}
Figure