Gaussian Weight Sampling for Scalable, Efficient and Stable Pseudo-Quantization Training

Myeonghwan Ahn, Sungjoo YooarXiv 2025 · DOI · arXiv

Abstract

Training Large Language Models (LLMs) in low-bit formats is essential for computational efficiency but often suffers from training instability and hardware incompatibility. We introduce Gaussian Weight Sampling (GaussWS), a novel pseudo-quantization training (PQT) method that replaces standard uniform noise with a "floating-point friendly" rounded Gaussian distribution. GaussWS enables stable training for models up to 1B parameters, matching or exceeding BF16 baselines with a negligible computational overhead of approximately 1.4%.

Introduction

The escalating cost of training LLMs has driven a shift toward lower-precision formats like FP8 or FP4. However, Fully Quantized Training (FQT) is notoriously brittle, often plagued by quantization-induced oscillations and requiring complex bit-precision searches. Pseudo-Quantization Training (PQT) offers a smoother alternative by adding pseudo-quantization noise (PQN) to high-precision weights, maintaining differentiability. Yet, existing PQT methods typically use uniform noise distributions that interact poorly with floating-point hardware, leading to "underflow" issues where small values are prematurely rounded to zero, causing training instability.

Methods: Gaussian Weight Sampling

GaussWS addresses these limitations through three primary innovations:

  1. Rounded Gaussian Noise: Instead of uniform noise, GaussWS employs a rounded normal distribution $\lfloor \mathcal{N}(0, 1)/2 \rceil$. This distribution is more compatible with floating-point (FP) hardware, supporting a wider range of effective bit-widths (up to 9-bit precision on a BF16 operator) compared to the 5-bit limit of uniform noise.
  2. Stochastic Precision Annealing: The Gaussian distribution has a high probability ($\approx 0.717$) of producing zero noise. This allows the model to frequently "see" high-precision weights, acting as an inherent annealing mechanism that stabilizes training against information loss in small-magnitude parameters.
  3. Efficient Bitwise Noise Generation: To minimize overhead, the authors developed a custom Triton kernel that generates Gaussian noise using bitwise AND and OR operations on random integer streams, bypassing slower traditional methods like the Box-Muller transform.
Schematic of the GaussWS method, illustrating the rounded Gaussian noise generation and its integration into the GEMM forward/backward passes.

Experimental Setup

The method was evaluated by pre-training models from scratch on major datasets:

  • OpenWebText: For GPT2-124M models.
  • C4 (Colossal Clean Crawled Corpus): For Llama2-134M and Llama2-1B models, with training extending up to 300 billion tokens.

Key Results and Insights

GaussWS demonstrates superior stability and efficiency compared to existing baselines.

Stability and Performance

In empirical tests, GaussWS successfully stabilized training in scenarios where standard BF16 baselines diverged. It consistently outperformed other PQT methods, such as DiffQ, in terms of final loss and convergence speed.

Training loss curves for GPT2-124M comparing AdamW, GaussWS, and DiffQ. GaussWS shows improved stability and lower final loss across different learning rates.

Computational Efficiency

On NVIDIA A100 GPUs, GaussWS added only 1.40% computational overhead for Llama2 training. Its custom kernel implementation is approximately 7x faster than standard PyTorch PQT implementations and 22% faster than DiffQ. Furthermore, it requires only 2 bytes per parameter in GPU memory, ensuring high scalability for large-scale models.

Module-Specific Instability

The researchers identified that different modules contribute to instability at different stages: the attention module is the primary source of early-stage instability (~30B tokens), while the feed-forward module tends to cause issues much later in the training process (~200B tokens).

Conclusion and Future Work

GaussWS provides a robust framework for low-bit LLM training by leveraging the properties of Gaussian noise. While currently focused on model weights, future work aims to extend this differentiable search to activations and gradients. Additionally, further optimization of hyperparameters, such as initial bit-width and weight decay for precision parameters, may yield even greater performance gains.

Summary updated

Abstract

Ever-growing scale of large language models (LLMs) is pushing for improved efficiency, favoring fully quantized training (FQT) over BF16. While FQT accelerates training, it faces consistency challenges and requires searching over an exponential number of cases, each needing over 200B tokens to ensure stability. Pseudo-quantization training (PQT) addresses the issues of FQT, although it is not well-studied. We explore the practical implications of PQT in detail and propose a noise distribution $R$ that is floating-point (FP)-friendly, with ideal properties including stochastic precision annealing. As a result, the proposed method serves as an effective theoretical foundation for low-precision FP parameters through PQT, utilizing efficient fake quantization via an addition and subsequent FP casting. We demonstrate that Gaussian weight sampling is (1) scalable: supports low-precision FP parameters down to FP6 and high-precision noise up to 9-bit with BF16 operator. The proposed method is (2) efficient: incurring computational overhead as low as 1.40\% on the A100 GPU in terms of Llama2 training tokens per second, and requiring 2 bytes per parameter in GPU memory. We demonstrate that PQT with Gaussian weight sampling is (3) stable: closely following or even surpassing performance of the BF16 baseline while pre-training GPT2 and Llama2 models with up to 1B parameters and 300B tokens.
BibTeX
@misc{ahn2025gaussian,
  author = {Myeonghwan Ahn and Sungjoo Yoo},
  title = {{Gaussian Weight Sampling for Scalable, Efficient and Stable Pseudo-Quantization Training}},
  year = {2025},
  eprint = {2505.11170},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.11170},
}
Figure