QuEST: Stable Training of LLMs with 1-Bit Weights and Activations

Andrei Panferov, Jiale Chen, Soroush Tabesh, Roberto L. Castro, Mahdi Nikdan, Dan Alistarh

TL;DR

QuEST introduces a novel Quantization-Aware Training (QAT) framework that enables stable LLM training down to ultra-low precisions, including 1-bit weights and activations. By combining a normalizing Hadamard Transform with a new "trust gradient estimator," QuEST pushes the Pareto-optimal frontier of LLM training down to 4-bits, delivering models that outperform standard 16-bit (BF16) baselines at a fraction of the computational and memory footprint.

What Problem Does This Paper Address?

As Large Language Models (LLMs) scale, their computational and memory demands become increasingly unsustainable. Quantization—reducing the bit-width of weights and activations—is a primary strategy to mitigate these costs. While Post-Training Quantization (PTQ) is widely used, it suffers severe accuracy degradation at ultra-low bit-widths (like 4-bit or lower).

Quantization-Aware Training (QAT), which simulates quantization during training, offers a potential solution by allowing the model to adapt to compression errors. However, training models with both low-bit weights and low-bit activations has historically been highly unstable. Recent studies even suggested that 8-bit precision was the fundamental "optimal" limit for QAT before accuracy collapsed. QuEST tackles this stability barrier head-on, aiming to make ultra-low bit-width QAT (down to 1-bit) both stable and highly accurate.

Methods

QuEST re-thinks both the forward and backward passes of QAT through two main innovations:

  1. Hadamard Preprocessing & Gaussian Fitting (Forward Pass): LLM activations typically exhibit severe "outlier channels" that make uniform quantization difficult. QuEST applies an orthogonal Hadamard Transform (HT) along the matrix-multiplication dimension. This mathematically "spreads out" the outlier energy, transforming the skewed distribution into a clean, sub-Gaussian distribution. QuEST then applies an analytically derived, Mean-Squared-Error (MSE)-optimal scaling factor to fit the grid, completely bypassing unstable backpropagation through the quantization scales.
  2. Trust Gradient Estimation (Backward Pass): Standard QAT relies on the Straight-Through Estimator (STE) to pass gradients through non-differentiable rounding operations. However, weights clipped during the forward pass introduce massive gradient bias. QuEST introduces a trust mask ($M_{\alpha^*}$) that zeroes out gradients for elements experiencing quantization errors larger than a specific threshold. Crucially, because the Hadamard Transform is inverted in the backward pass, these masked weights aren't permanently frozen; gradient flow is dynamically distributed across all standard-domain parameters.

Datasets

The authors evaluated QuEST by pre-training Llama-family models (ranging from 30M to 1.6B parameters) from scratch using:

  • C4 (Colossal Clean Crawled Corpus): Used for primary pre-training and scaling law evaluations, with models trained on up to 160 billion tokens.
  • Commonsense Reasoning Benchmarks: For zero-shot evaluation, the models were tested on HellaSWAG, ARC (Easy and Challenge), PiQA, and Winogrande.

Key Insights and Ideas

  • Outliers are Gradient Poison: The core reason low-bit QAT fails isn't the average quantization noise, but the extreme gradient bias caused by a small subset of outlier entries.
  • Hadamard as a Stabilizer: The Hadamard Transform acts as a mathematical blender. By smoothing out outliers, it ensures that the "trust mask" doesn't permanently lock out important weights, maintaining high gradient alignment throughout deep networks.
Figure 2. Gradient alignment comparison for a 30M Llama model after training on 2.7B tokens in 8-bit precision.
  • Redefining the Pareto Frontier: In the "overtraining" regime (where models are trained far past the Chinchilla-optimal token-to-parameter ratio to minimize deployment costs), 4-bit precision emerges as the absolute optimal choice for maximizing accuracy per unit of runtime compute.

Key Results

  • Stable 1-Bit Training: QuEST achieved stable convergence for 1-bit weights and 1-bit activations (W1A1), establishing predictable scaling laws across all tested precisions.
Figure 1. The scaling law induced by QuEST when training Llama- family models from 30 to 1.6B parameters on C4, with quan- tized weights and activations from 1 to 4 bits, in the 100 to- kens/parameter regime (harder compression uses proportionally more data at fixed memory). QuEST allows for stable training at 1- bit weights and activations (W1A1), and the QuEST W4A4 model is Pareto-dominant relative to BF16, with lower loss at lower size.
  • Pareto Dominance at 4-Bits: The QuEST W4A4 model consistently outperformed standard BF16 models of equivalent validation loss while being nearly 4x smaller in memory footprint.
  • Superior Efficiency: When measuring parameter efficiency relative to bit-width, QuEST INT4 achieved the highest efficiency factor among all tested formats, including FP4 and sparse alternatives.
Figure 3. Illustration of the efficiency factors eff(P)/P, arising from our analysis, for different numerical precisions P, formats (INT, FP, INT+sparse) and methods. Higher is better. QuEST INT4 appears to have the highest efficiency.
  • Hardware Speedups: The authors developed custom Triton and CUTLASS GPU kernels to support QuEST. For a 7B parameter model, these kernels delivered 2.3× to 3.9× per-layer speedups during inference on a single RTX 4090 GPU compared to standard BF16 execution.

Limitations and Future Work

While QuEST demonstrates impressive stability and scaling behavior, the authors note a few limitations:

  • Model Scale: The empirical validation was performed on models up to 1.6B parameters. Testing QuEST's scaling behavior on truly massive models (e.g., 70B+ parameters) remains an open direction.
  • Architecture Scope: The current implementation focuses exclusively on decoder-only Transformer architectures. Extending the framework to encoder-decoder models or alternative architectures (like state-space models) is left for future work.

Summary updated

Abstract

One approach to reducing the massive costs of large language models (LLMs) is the use of quantized or sparse representations for training or deployment. While post-training compression methods are very popular, the question of obtaining even more accurate compressed models by directly training over such representations, i.e., Quantization-Aware Training (QAT), is still open: for example, a recent study (arXiv:2411.04330) put the "optimal" bit-width at which models can be trained using QAT, while staying accuracy-competitive with standard FP16/BF16 precision, at 8-bits weights and activations. We advance this state-of-the-art via a new method called QuEST, for which we demonstrate optimality at 4-bits and stable convergence as low as 1-bit weights and activations. QuEST achieves this by improving two key aspects of QAT methods: (1) accurate and fast quantization of the (continuous) distributions of weights and activations via Hadamard normalization and MSE-optimal fitting; (2) a new trust gradient estimator based on the idea of explicitly minimizing the error between the noisy gradient computed over quantized states and the "true" (but unknown) full-precision gradient. Experiments on Llama-type architectures show that QuEST induces stable scaling laws across the entire range of hardware-supported precisions, and can be extended to sparse representations. We provide GPU kernel support showing that models produced by QuEST can be executed efficiently. Our code is available at https://github.com/IST-DASLab/QuEST.
BibTeX
@inproceedings{panferov2025quest,
  author = {Andrei Panferov and Jiale Chen and Soroush Tabesh and Roberto L. Castro and Mahdi Nikdan and Dan Alistarh},
  title = {{QuEST: Stable Training of LLMs with 1-Bit Weights and Activations}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=I0Ux2nAN6u},
}
Figure