Towards Efficient Optimizer Design for LLM via Structured Fisher Approximation with a Low-Rank Extension

Wenbo Gong, Meyer Scetbon, Chao Ma, Edward Meeds

The Memory Bottleneck in LLM Training

Training Large Language Models (LLMs) is notoriously memory-intensive. Surprisingly, a massive chunk of this memory is hogged not by the model weights themselves, but by the optimizer. The industry standard, Adam, triples the parameter memory footprint because it stores two internal moving average states for every single parameter.

While memory-efficient optimizers exist (like standard SGD), they usually fail to train LLMs effectively. Recently, researchers have proposed various low-rank or state-free optimizers (like GaLore, Muon, or Adafactor) to bridge this gap. However, these are often designed using ad-hoc heuristics. There is currently no systematic, principled framework to design optimizers that perfectly balance memory efficiency with fast convergence. This paper aims to provide that exact framework.

A Unifying Viewpoint: Structured FIM Approximation

The authors propose that many state-of-the-art optimizers can be understood through a single mathematical lens: structured Fisher Information Matrix (FIM) approximation. By minimizing the difference (specifically, the Frobenius norm) between the true FIM and a simplified matrix structure, researchers can derive different optimizers.

Under this unifying viewpoint, popular optimizers aren't as fundamentally different as they seem. The authors prove that Adam is simply a purely diagonal FIM approximation; Shampoo is a Kronecker product approximation; and Muon/Whitening are simple block diagonal approximations.

Taxonomy of optimizer structures based on FIM approximation

Building on this insight, the authors propose two distinct design routes resulting in two new optimizers: RACS and Alice.

1. Balancing Generality and Efficiency: RACS

The first approach involves designing a structure that generalizes gradient normalization by scaling both rows and columns simultaneously. The resulting optimizer, Row and Column Scaled SGD (RACS), requires almost no optimizer states. It achieves SGD-like memory efficiency while delivering much stronger convergence properties.

2. The Low-Rank Extension Framework: Alice

For more complex FIM structures that are too memory-heavy to compute directly, the authors propose a three-step low-rank extension framework. They apply this framework to a generalized version of Adam to create Alice (Adaptive low-dimensional subspace estimation).

The three-step low-rank extension framework used in Alice

Alice relies on three core mechanisms:

  • Low-rank tracking: Compressing the optimizer states to save memory.
  • Subspace switching: Periodically swapping out leading eigenvectors for random complement vectors. This forces the optimizer to explore new spaces.
  • Compensation: Mathematically adding back the gradient information lost during the low-rank projection step.

The "Top Eigenvector" Trap: Standard low-rank optimizers assume you should only track the top eigenvectors of the gradient. The authors prove mathematically that doing this causes the optimizer to get stuck, as it ignores emerging directions. Subspace switching—deliberately mixing less important directions with the leading ones—is critical for better performance. In fact, the popular memory-efficient optimizer GaLore is revealed to be a restricted, special case of Alice where tracking, switching, and compensation are disabled.

Experimental Setup and Key Results

The authors validated their optimizers by pre-training LLaMA models (ranging from 60M up to 1.3B parameters) on the C4 dataset, and fine-tuning RoBERTa on the GLUE benchmark for downstream tasks.

The results demonstrate significant improvements in both speed and memory:

  • Massive Speedups: Alice achieves more than 2x faster convergence (both in training steps and wall-clock time) compared to standard Adam.
  • Drastic Memory Reduction: Alice uses roughly 55% of Adam's memory (similar to GaLore). Meanwhile, RACS achieves SGD-like memory footprints, requiring only $O(m+n)$ memory overhead compared to Adam's $O(mn)$.
  • Outperforming Baselines: On a 1.3B parameter LLaMA model, both Alice and RACS consistently outperformed existing memory-efficient baselines, including GaLore, Fira, Apollo, and Muon, achieving lower evaluation perplexity.
  • Punching Above Their Weight: In a controlled setup, a 1B LLaMA model trained with Alice/RACS consistently outperformed a 7B model trained with baseline memory-efficient optimizers at various step checkpoints, highlighting the raw efficiency of the optimization path.

Limitations and Future Work

The authors acknowledge that this paper is a starting point rather than a complete recipe book. Specifically, they do not provide a complete theoretical convergence treatment for all optimizers under this FIM framework, though they do prove continuous-time convergence for Alice.

For future work, they suggest applying their low-rank extension framework to even more complex optimizers (like SOAP). They also note that figuring out how to incorporate specific model architectures and downstream task structures directly into the FIM approximation design is a highly promising research direction.

Summary updated

Abstract

Designing efficient optimizers for large language models (LLMs) with low-memory requirements and fast convergence is an important and challenging problem. This paper makes a step towards the systematic design of such optimizers through the lens of structured Fisher information matrix (FIM) approximation. We show that many state-of-the-art efficient optimizers can be viewed as solutions to FIM approximation (under the Frobenius norm) with specific structural assumptions. Building on these insights, we propose two design recommendations of practical efficient optimizers for LLMs, involving the careful selection of structural assumptions to balance generality and efficiency, and enhancing memory efficiency of optimizers with general structures through a novel low-rank extension framework. We demonstrate how to use each design approach by deriving new memory-efficient optimizers: Row and Column Scaled SGD (RACS) and Adaptive low-dimensional subspace estimation (Alice). Experiments on LLaMA pre-training (up to 1B parameters) validate the effectiveness, showing faster and better convergence than existing memory-efficient baselines and Adam with little memory overhead. Notably, Alice achieves better than 2x faster convergence over Adam, while RACS delivers strong performance on the 1B model with SGD-like memory.
BibTeX
@inproceedings{gong2026towards,
  author = {Wenbo Gong and Meyer Scetbon and Chao Ma and Edward Meeds},
  title = {{Towards Efficient Optimizer Design for LLM via Structured Fisher Approximation with a Low-Rank Extension}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=KUFZXdem5R},
}
Figure