Stabilizing Native Low-Rank LLM Pretraining

Paul Janson, Edouard Oyallon, Eugene Belilovsky

TL;DR

Researchers have finally found a stable "recipe" for training Large Language Models (LLMs) from scratch using exclusively low-rank factorized weights. By introducing Spectron, a method that regulates the spectral norm of weight updates, they achieved performance parity with dense models at equal compute while using ~42% fewer parameters, effectively creating more "inference-optimal" models.

What Problem Does This Paper Address?

As LLMs grow, their memory and computational requirements become massive bottlenecks. Low-rank factorization—representing a large weight matrix $W$ as the product of two smaller matrices $A$ and $B$ ($W = AB^\top$)—is a classic way to compress models.

However, "native" low-rank pretraining (training from scratch without any full-rank weights) is notoriously unstable. It often suffers from catastrophic loss spikes and divergence. To get around this, previous state-of-the-art methods used "crutches," such as keeping auxiliary full-rank weights during the first half of training or only factorizing specific layers. These workarounds increase memory overhead and complexity, defeating the primary purpose of low-rank training. This paper tackles the root cause of that instability to enable pure, end-to-end low-rank training.

Methods

The authors identify the culprit of instability: uncontrolled growth in the spectral norm (the largest singular value) of the weight matrix updates. Because $W = (\lambda A)(\frac{1}{\lambda} B)$ for any $\lambda > 0$, the individual factors can grow or shrink wildly while representing the same product, leading to "exploding" activations.

To fix this, they introduce Spectron (Spectral renormalization with orthogonalization). Spectron consists of two main parts:

  1. Gradient Orthogonalization: Using Newton-Schulz iterations to ensure that the updates to factors $A$ and $B$ are orthogonal, which helps stabilize the geometry of the update.
  2. Spectral Renormalization: A novel adaptive scaling step. It calculates a "local constraint radius" based on the current spectral norms of factors $A$ and $B$. This ensures that the resulting composite update $\Delta W$ stays within a stable bound defined by the learning rate.

Crucially, Spectron adds less than 1% computational overhead, making it significantly more efficient than previous "self-guided" methods that added ~25% more FLOPs.

Datasets

The models were pretrained on FineWeb, a high-quality large-scale web dataset. For evaluation, the authors used a 100M token validation set and several standard downstream benchmarks:

  • HellaSwag: Common sense reasoning.
  • PIQA: Physical interaction QA.
  • ARC-Easy: Elementary science questions.

Key Insights and Ideas

  • The Scaling Invariance Trap: The fundamental issue in low-rank training is that independent updates to $A$ and $B$ provide no mechanism to control the spectral properties of the resulting $W$. Spectron provides that missing link.
  • Inference-Optimal vs. Training-Optimal: Dense models are often trained to be "Chinchilla-optimal" (balancing model size and data for a fixed training budget). This paper shows that low-rank models shift the frontier: they favor even smaller model sizes trained on more tokens. This results in models that are much cheaper to run at inference time for the same level of "intelligence."
  • Predictable Scaling: Low-rank transformers follow predictable power-law scaling, just like dense ones. The authors derived new scaling laws specifically for low-rank architectures, showing they scale slightly more "conservatively" than dense models but offer better parameter efficiency.

Key Results

  • Performance Parity: A 454M parameter low-rank model trained with Spectron matched the validation loss of a 780M parameter dense model when given the same total training compute (FLOPs).
  • Parameter Reduction: The method consistently achieves dense-level performance with a ~42% reduction in parameters.
  • Superior Stability: Unlike naive AdamW, which diverges at higher learning rates in low-rank settings, Spectron remains stable, allowing for faster convergence and better final perplexity.
  • Downstream Gains: On benchmarks like HellaSwag and PIQA, Spectron-trained models consistently outperformed both naive low-rank baselines and "self-guided" methods across all tested scales (94M to 454M parameters).

Limitations and Future Work

  • Architecture Scope: The study focused on Llama-style Transformer architectures. While the principles likely apply elsewhere, the authors haven't yet tested them on multimodal or non-Transformer architectures.
  • Communication Overhead: While Spectron reduces memory and compute, the authors note that future work is needed to develop communication strategies specifically tailored for distributed training of factorized layers to further reduce overhead.
  • Extreme Compression: The authors found that while a rank ratio of 0.25 works well, very aggressive compression (e.g., 0.125) still leads to degraded performance, suggesting there is a lower bound on how much we can factorize during pretraining.

Summary updated

Abstract

Foundation models have achieved remarkable success, yet their growing parameter counts pose significant computational and memory challenges. Low-rank factorization offers a promising route to reduce training and inference costs, but the community lacks a stable recipe for training models from scratch using exclusively low-rank weights while matching the performance of the dense model. We demonstrate that Large Language Models (LLMs) can be trained from scratch using exclusively low-rank factorized weights for all non-embedding matrices without auxiliary "full-rank" guidance required by prior methods. While native low-rank training often suffers from instability and loss spikes, we identify uncontrolled growth in the spectral norm (largest singular value) of the weight matrix update as the dominant factor. To address this, we introduce Spectron: Spectral renormalization with orthogonalization, which dynamically bounds the resultant weight updates based on the current spectral norms of the factors. Our method enables stable, end-to-end factorized training with negligible overhead. Finally, we establish compute-optimal scaling laws for natively low-rank transformers, demonstrating predictable power-law behavior and improved inference efficiency relative to dense models.
BibTeX
@inproceedings{janson2026stabilizing,
  author = {Paul Janson and Edouard Oyallon and Eugene Belilovsky},
  title = {{Stabilizing Native Low-Rank LLM Pretraining}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=kb5wjkqu8q},
}
Figure