µnit Scaling: Simple and Scalable FP8 LLM Training

Abhay Gupta, Davis Blalock, Saaketh Narayan, Mansheej Paul

Introduction

Training Large Language Models (LLMs) requires immense computational resources. Transitioning from 16-bit (BF16) to 8-bit floating-point (FP8) precision offers massive theoretical speedups and memory savings. However, the reduced numerical precision of FP8 makes training notoriously unstable. To address this, researchers have introduced µnit Scaling (µS), a novel framework that enables stable, high-performance LLM training in FP8 without the need for complex dynamic scaling. By combining variance-preserving architectural modifications with a simplified version of µ-Parametrization, µS allows researchers to tune hyperparameters on tiny models and seamlessly transfer them to models with over 13 billion parameters, achieving up to a 33% training speedup over standard 16-bit methods.

The Problem: The FP8 Precision Bottleneck

The primary challenge with FP8 is its extremely narrow numerical range. If tensor values "overflow" (become too large) or "underflow" (become too small), the training process collapses.

To mitigate this, current state-of-the-art libraries—such as NVIDIA’s TransformerEngine—rely on dynamic scaling. This approach calculates scaling factors for every tensor on the fly during training. While effective at preventing numerical collapse, dynamic scaling introduces computational overhead, complicates distributed training setups, and fails to solve the "hyperparameter headache." Optimal learning rates and initialization settings for a small model often fail completely when scaled up to a larger one. The authors of µnit Scaling sought a statically scaled alternative that is simple to implement, computationally efficient, and stable at massive scales.

Methodology: The µnit Scaling Framework

µnit Scaling (µS) is built on a first-principles analysis of how variance flows through a Transformer network. To keep tensors within the representable range of FP8 throughout an entire training run, the authors propose several key interventions:

  • Unit Scaling: Instead of calculating dynamic scales, µS uses static constants based on the "fan-in" of network layers and initializes weights to unit variance.
  • Square-Root Softmax Attention: The authors discovered that standard causal attention naturally causes variance to drop as a sequence progresses because later tokens attend to a larger number of previous tokens. To preserve variance, they introduce a modified attention mechanism that takes the square root of the softmax scores.
  • Res-Post-LayerNorm: Standard initialization theory assumes tokens are independent, but real-world text is highly correlated. This correlation causes variance to explode in standard "Pre-LayerNorm" architectures. Moving the LayerNorm to the end of the residual branch (Res-Post-LN) prevents this variance drift.
  • Fixed Residual Modification: Skip connections are managed using a weighted sum ($a^2 + b^2 = 1$) with a fixed coefficient $\tau$ that scales based on the depth of the model.
  • Simplified µ-Parametrization (µP): By integrating µP rules, µS enables "zero-shot" hyperparameter transfer. Researchers can find the optimal learning rate on a small proxy model (e.g., width 256) and apply it directly to a massive model (e.g., width 5120). Previous attempts to combine unit scaling and µP required managing up to seven hyperparameters; µS reduces this to just three ($\eta, \lambda, \tau$).

Experimental Setup

The authors validated µS by training decoder-only Transformers ranging from 1B to 13B parameters on a compute-optimal token budget (roughly 20 tokens per parameter). All hidden linear layer computations were performed in FP8.

Model quality was evaluated using the comprehensive Databricks Model Gauntlet, which includes over 30 benchmarks spanning Reasoning and Knowledge (ARC, HellaSwag, PIQA, MMLU-style tasks), Language Understanding (SQuAD, Winogrande, CoQA), and Coding/Math (GSM8K).

Key Results

The implementation of µnit Scaling yielded significant improvements across speed, stability, and model quality:

  • Training Speed: µS in FP8 is 25–33% faster than standard BF16 training. Furthermore, it is 1–6% faster than NVIDIA’s TransformerEngine because it completely eliminates the overhead of dynamic scaling.
  • Unprecedented Stability: The authors successfully trained models up to 13B parameters in FP8. In contrast, the standard baseline using TransformerEngine experienced frequent loss spikes and failed to converge at the 13B scale.
  • Maintained Quality: Across the Model Gauntlet benchmarks, µS models achieved equal or slightly better accuracy compared to higher-precision BF16 baselines.
  • Inference Readiness: Because the model is natively trained in FP8, it avoids the quantization degradation typically observed when converting a BF16 model to 8-bit for deployment. It is essentially "born" ready for W8A8 inference.

Limitations and Future Directions

While µnit Scaling represents a major leap forward for efficient training, the authors note a few limitations. Popular activation functions like GELU and SiLU can cause significant "underflow" (values flushing to zero) in FP8. While ReLU performs better numerically, it results in slightly higher final loss. Designing FP8-specific activation functions remains a promising area for future research.

Additionally, this study focused exclusively on decoder-only Transformers. Applying µS to encoder-decoder models or novel architectures requires further validation. Finally, while stabilizing FP8 training at 13B parameters is a significant milestone, testing µS at the 70B or 400B+ parameter scale will be the ultimate test to cement it as the definitive training methodology for next-generation frontier models.

Summary updated

Abstract

Large Language Model training with 8-bit floating point (FP8) formats promises significant efficiency improvements, but reduced numerical precision makes training challenging. It is currently possible to train in FP8 only if one is willing to tune various hyperparameters, reduce model scale, or accept the overhead of computing dynamic scale factors. We demonstrate simple, scalable FP8 training that requires no dynamic scaling factors or special hyperparameters, even at large model sizes. Our method, μnit Scaling (μS), also enables simple hyperparameter transfer across model widths, matched numerics across training and inference, and other desirable properties. μnit Scaling is straightforward to implement, consisting of a set of minimal interventions based on a first-principles analysis of common transformer operations. We validate our method by training models from 1B to 13B parameters, performing all hidden linear layer computations in FP8. We achieve quality equal to higher precision baselines while also training up to 33% faster.
BibTeX
@inproceedings{gupta2025nit,
  author = {Abhay Gupta and Davis Blalock and Saaketh Narayan and Mansheej Paul},
  title = {{µnit Scaling: Simple and Scalable FP8 LLM Training}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=qOLjAhxZgm},
}
Figure