Abstract
Training Large Language Models (LLMs) is often bottlenecked by the massive memory required for optimizer states. This paper introduces a systematic framework for designing efficient optimizers by viewing them as approximations of the Fisher Information Matrix (FIM). Using this lens, the authors develop two new optimizers: RACS, which offers SGD-like memory with much better convergence, and Alice, a low-rank optimizer that converges over 2x faster than Adam with significantly less memory overhead.
Introduction
In the world of LLMs, "memory is king." While standard adaptive optimizers like Adam are the industry workhorse due to their fast convergence, they come with a heavy "memory tax." Adam requires storing two internal states (moving averages of gradients and their squares) for every single parameter in the model, effectively tripling the memory needed for weights. As models scale to billions of parameters, this overhead limits batch sizes and increases hardware requirements. The challenge is to find the "Goldilocks" optimizer: one that converges as fast as Adam but uses memory closer to SGD.
Methods: A Unified Framework for Optimizer Design
The authors propose that the design of efficient optimizers can be unified through Structured FIM Approximation. They demonstrate that many existing optimizers (like Adam, Shampoo, and SOAP) are actually different ways of approximating the Fisher Information Matrix—a matrix that captures the local geometry of the loss surface—under specific structural assumptions.
Building on this theory, they introduce two design paths:
1. RACS (Row and Column Scaled SGD)
This optimizer is designed by choosing a structural assumption that balances complexity and efficiency. It scales the rows and columns of the gradient matrix simultaneously. This requires only $O(m+n)$ memory (where $m$ and $n$ are weight matrix dimensions) compared to Adam’s $O(mn)$.
2. Alice (Adaptive low-dimensional subspace estimation)
This is a more sophisticated low-rank extension framework consisting of three novel steps:
- Low-rank Tracking: Projecting gradients into a smaller subspace to save memory on optimizer states.
- Subspace Switching: Periodically mixing the "leading" basis with random components from the complement space to ensure the optimizer doesn't get stuck in a stale subspace.
- Compensation: A critical step that adds back a simplified version of the discarded information, ensuring the update remains "full-rank" and maintains high convergence speed.
Evaluation and Key Results
The researchers evaluated their optimizers by pre-training LLaMA models from scratch on the C4 (Colossal Clean Crawled Corpus) dataset. Model scales ranged from 60M to 1.3B parameters.
- Speedup: Alice achieved more than 2x faster convergence compared to Adam across various model sizes. In wall-clock time, Alice reached Adam's final evaluation loss in roughly 1/4th of the time (3.8 days vs 15 days on a specific 8x A100 setup).
- Memory Efficiency: RACS delivered strong performance on the 1B model while maintaining SGD-like memory footprints.
- Performance Scaling: Remarkably, a 1B parameter model trained with Alice reached an evaluation perplexity that was on-par or better than a 7B parameter model trained with other state-of-the-art memory-efficient baselines.
- Throughput: Despite the additional steps in Alice (switching and compensation), the actual training throughput remained high, only dropping about 15% compared to Adam.
Conclusion
The unified view of optimizer design as FIM approximation provides a powerful tool for developing next-generation training algorithms. RACS and Alice demonstrate that it is possible to significantly reduce memory overhead without sacrificing—and in some cases, even improving—convergence speed for large-scale language models. Future work may explore applying these low-rank frameworks to even more general structures like those used in the SOAP optimizer.
Summary updated