Abstract
Training Large Language Models (LLMs) in low-bit formats is essential for computational efficiency but often suffers from training instability and hardware incompatibility. We introduce Gaussian Weight Sampling (GaussWS), a novel pseudo-quantization training (PQT) method that replaces standard uniform noise with a "floating-point friendly" rounded Gaussian distribution. GaussWS enables stable training for models up to 1B parameters, matching or exceeding BF16 baselines with a negligible computational overhead of approximately 1.4%.
Introduction
The escalating cost of training LLMs has driven a shift toward lower-precision formats like FP8 or FP4. However, Fully Quantized Training (FQT) is notoriously brittle, often plagued by quantization-induced oscillations and requiring complex bit-precision searches. Pseudo-Quantization Training (PQT) offers a smoother alternative by adding pseudo-quantization noise (PQN) to high-precision weights, maintaining differentiability. Yet, existing PQT methods typically use uniform noise distributions that interact poorly with floating-point hardware, leading to "underflow" issues where small values are prematurely rounded to zero, causing training instability.
Methods: Gaussian Weight Sampling
GaussWS addresses these limitations through three primary innovations:
- Rounded Gaussian Noise: Instead of uniform noise, GaussWS employs a rounded normal distribution $\lfloor \mathcal{N}(0, 1)/2 \rceil$. This distribution is more compatible with floating-point (FP) hardware, supporting a wider range of effective bit-widths (up to 9-bit precision on a BF16 operator) compared to the 5-bit limit of uniform noise.
- Stochastic Precision Annealing: The Gaussian distribution has a high probability ($\approx 0.717$) of producing zero noise. This allows the model to frequently "see" high-precision weights, acting as an inherent annealing mechanism that stabilizes training against information loss in small-magnitude parameters.
- Efficient Bitwise Noise Generation: To minimize overhead, the authors developed a custom Triton kernel that generates Gaussian noise using bitwise
ANDandORoperations on random integer streams, bypassing slower traditional methods like the Box-Muller transform.
Experimental Setup
The method was evaluated by pre-training models from scratch on major datasets:
- OpenWebText: For GPT2-124M models.
- C4 (Colossal Clean Crawled Corpus): For Llama2-134M and Llama2-1B models, with training extending up to 300 billion tokens.
Key Results and Insights
GaussWS demonstrates superior stability and efficiency compared to existing baselines.
Stability and Performance
In empirical tests, GaussWS successfully stabilized training in scenarios where standard BF16 baselines diverged. It consistently outperformed other PQT methods, such as DiffQ, in terms of final loss and convergence speed.
Computational Efficiency
On NVIDIA A100 GPUs, GaussWS added only 1.40% computational overhead for Llama2 training. Its custom kernel implementation is approximately 7x faster than standard PyTorch PQT implementations and 22% faster than DiffQ. Furthermore, it requires only 2 bytes per parameter in GPU memory, ensuring high scalability for large-scale models.
Module-Specific Instability
The researchers identified that different modules contribute to instability at different stages: the attention module is the primary source of early-stage instability (~30B tokens), while the feed-forward module tends to cause issues much later in the training process (~200B tokens).
Conclusion and Future Work
GaussWS provides a robust framework for low-bit LLM training by leveraging the properties of Gaussian noise. While currently focused on model weights, future work aims to extend this differentiable search to activations and gradients. Additionally, further optimization of hyperparameters, such as initial bit-width and weight decay for precision parameters, may yield even greater performance gains.
Summary updated