Abstract
Training Large Language Models (LLMs) in 8-bit floating point (FP8) precision offers significant throughput advantages but is traditionally hampered by numerical instability and the complexity of dynamic scaling. We present μnit Scaling (μS), a framework that combines unit variance principles with maximal update parametrization (μP). By ensuring tensors maintain unit variance through static scaling and architectural modifications, μS enables stable FP8 training up to 13B parameters without the overhead of dynamic scaling. Our results demonstrate a 33% throughput increase over BF16 and seamless hyperparameter transfer across model scales.
Introduction
The computational cost of training state-of-the-art LLMs has driven the industry toward lower-precision numerical formats. Moving from 16-bit (BF16) to 8-bit (FP8) precision offers a massive theoretical speedup; however, FP8 possesses a very narrow numerical range. In standard training regimes, activations or gradients frequently "overflow" to infinity or "underflow" to zero, leading to training divergence.
To mitigate this, current state-of-the-art methods like NVIDIA’s TransformerEngine employ dynamic scaling, which calculates the maximum value of every tensor at every step to shift them into a representable range. This approach introduces computational overhead and complicates distributed training. μnit Scaling (μS) provides a "set-it-and-forget-it" alternative that works at scale without the need for dynamic scaling or extensive hyperparameter "babysitting."
Methods
μnit Scaling relies on minimal interventions based on a first-principles analysis of transformer operations to maintain numerical stability.
1. Unit Variance Maintenance
The fundamental principle of μS is ensuring that every weight, activation, and gradient tensor maintains a variance of approximately 1. By keeping the "signal strength" constant throughout the network, the numerical values are significantly less likely to exit the narrow FP8 representable range.
2. Static Scaling Factors
Instead of calculating scales on the fly, μS utilizes fixed, mathematically derived constants (e.g., $1/\sqrt{fan\_in}$) applied to linear layers. This removes the need to store and communicate "max" values across GPUs, enhancing physical training speed.
3. Architectural Interventions
- Res-Post-LayerNorm: The authors identified that standard causal self-attention causes the variance of the output to decrease as it progresses along a sequence, making late-sequence tokens difficult to represent. Moving LayerNorm to the end of residual branches directly addresses this variance collapse.
- Fixed Residual Modification: To prevent the residual stream's variance from growing uncontrollably as the network depth increases, μS employs a weighted sum ($a^2 + b^2 = 1$) for skip connections.
4. Hyperparameter Transfer via μP
μS integrates maximal update parametrization (μP) such that unit scaling and μP work in tandem. This allows researchers to find optimal hyperparameters (such as learning rate) on a small proxy model and transfer them directly to models that are significantly wider (up to 20x) without performance degradation.
Experimental Setup
Models ranging from 1B to 13B parameters were trained on standard natural language datasets using compute-optimal token budgets (approximately a 20:1 token-to-parameter ratio). To ensure that low-precision training did not degrade model quality, the authors utilized the Databricks Model Gauntlet, a suite of over 30 benchmarks including ARC, HellaSwag, MMLU, and GSM8K.
Results
Throughput and Efficiency
μS demonstrated a 25–33% speedup over standard BF16 training. Furthermore, it was 1–6% faster than NVIDIA’s TransformerEngine, confirming that the removal of dynamic scaling provides a tangible hardware efficiency gain.
Stability at Scale
While standard FP8 training often suffers from loss spikes and divergences at the 13B parameter scale, μS models trained smoothly. The framework successfully transferred learning rates from a model with a width of 256 to one with a width of 5120 (a 20x increase) with zero-shot success.
Model Quality
In the Model Gauntlet evaluations, μS models achieved quality parity with, or in some cases exceeded, higher-precision BF16 baselines across nearly all tasks. This suggests that the constraints imposed by unit scaling do not inhibit the model's representative power.
Discussion and Limitations
While μS represents a significant step forward for FP8 training, some areas remain for future exploration. The authors noted that while ReLU is effective at preventing underflow, GELU still yields lower overall loss; finding or designing activation functions specifically for FP8 remains a promising path. Additionally, while the 13B scale is a significant milestone, the efficacy of μS at the trillion-parameter scale and its interaction with specialized initialization schemes (such as those for Mixture-of-Experts models) are yet to be tested.
Summary updated