Low-Bit Quantization for Dense and Mixture of Experts Models

This direction is split into other directions: sparse architecture quantization. Its last version stays readable and citable here.

Version 1 of 140 papers41 conceptsfrozen 2026-10-04

Deploying Large Language Models (LLMs) is primarily a battle against memory bandwidth. As models scale to hundreds of billions of parameters, the bottleneck shifts from raw computation to the speed at which weights can be moved from memory to the processor. Quantization—the process of representing weights and activations in lower-precision formats like 4-bit or even 2-bit—is the primary lever for overcoming this bottleneck.

While 8-bit quantization is now standard, the research community is currently focused on two frontiers: pushing precision to ultra-low "sub-bit" levels and ensuring these techniques generalize from standard dense Transformers to sparse architectures like Mixture-of-Experts (MoE).

The Geometry of Quantization Error

The fundamental challenge of quantization is that neural network weights and activations are not distributed uniformly. Most values cluster near zero, but a small number of "outliers" possess extreme magnitudes. In a naive linear quantization scheme, these outliers force the quantization grid to stretch so wide that the precision for the majority of values is lost, leading to a "functional collapse" of the model.

To solve this, researchers like Dan Alistarh and Maximilian L. Croci have proposed QuaRot, which uses Hadamard transformations to "rotate" the feature space. This mathematically smears the outliers across all dimensions, making the distribution more uniform and easier to quantize without losing information. Similarly, Boris van Breugel and Markus Nagel introduced FPTQuant, which uses function-preserving transforms to reshape activation distributions before they are compressed.

The Mixture of Experts Frontier

A recurring theme across recent literature is the unique behavior of Mixture-of-Experts (MoE) models under quantization. Unlike dense models, MoEs use a routing mechanism to send tokens to specific "experts." This sparsity introduces a new variable: how does quantization noise interact with the router?

Work by Young Jin Kim in FineQuant suggests that MoE architectures might actually be more resilient to weight quantization than dense models, potentially allowing them to maintain performance at 3-bit levels. However, this is balanced by findings from Krishnasai Addala, who noted that MoEs can exhibit high sensitivity in complex reasoning tasks. The consensus across papers like HIGGS and OSTQuant is that while dense quantization is maturing, the interaction between sparse routing and ultra-low precision remains an open challenge.

Moving Beyond Post-Training Quantization

Most early quantization was "Post-Training" (PTQ), where a pre-trained model is compressed using a small calibration set. However, as we push toward 4-bit and 2-bit limits, the "quantization gap"—the performance difference between the full-precision and compressed model—widens.

To close this gap, researchers are moving toward Quantization-Aware Training (QAT). Brian Chmiel and Ron Banner demonstrated in FP4 All the Way that it is possible to train LLMs from scratch using 4-bit floating-point precision for weights, activations, and gradients. This is supported by large-scale industrial efforts from NVIDIA, which successfully pretrained a 12B parameter model using NVFP4, achieving parity with 8-bit baselines.

Hardware-Algorithm Co-Design

The final piece of the puzzle is ensuring that these low-bit formats actually translate to speedups on real hardware. Standard GPUs are optimized for 16-bit or 8-bit arithmetic; 3-bit or sub-bit formats often require custom kernels to avoid the overhead of "dequantizing" back to higher precision during computation.

Zeyu Li and Zhenheng Tang addressed this in AnTKV by using token-aware vector quantization for the KV cache, enabling context lengths to scale to 840K tokens. On the architectural side, Yuzong Chen proposed P³-LLM, an accelerator that uses hybrid numerical formats (W4A8KV4P8) to match specific operands to the most efficient hardware units, achieving nearly 5x speedups over traditional memory-constrained designs.

The Role of Training Dynamics

A surprising discovery by Albert Catalan-Tatjer and Jonas Geiping is that a model's robustness to quantization is determined long before the quantization process begins. In Training Dynamics Impact Post-Training Quantization Robustness, they show that the sharpness of the loss landscape—influenced by learning rate decay and weight averaging during pretraining—directly dictates how much a model will degrade when compressed. This suggests that the next generation of LLMs will be "born" to be quantized, with training recipes specifically tuned for downstream efficiency.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_low_bit_quantization_for_dense_and_mixtu_3d323a3d,
  title = {Low-Bit Quantization for Dense and Mixture of Experts Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/low-bit-quantization-for-dense-and-mixture-of-experts-models/rdr_3d323a3d9f1e4fceb22f1be4a223477e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_low_bit_quantization_for_dense_and_mixtu_3d323a3d_v1,
  title = {Low-Bit Quantization for Dense and Mixture of Experts Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/low-bit-quantization-for-dense-and-mixture-of-experts-models/rdr_3d323a3d9f1e4fceb22f1be4a223477e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}