A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms

Ruihao Gong, Yifu Ding, Zining Wang, Chengtao Lv, Xingyu Zheng, Jinyang Du, Haotong Qin, Jinyang Guo, Michele Magno, Xianglong Liu

TL;DR

Low-bit quantization is essential for deploying massive Large Language Models (LLMs) on resource-constrained hardware. This comprehensive survey systematically maps the latest breakthroughs in low-bit LLMs, spanning novel data formats, hardware system implementations, and algorithmic strategies for both training and inference.

What Problem Does This Paper Address?

While LLMs have revolutionized natural language processing, their massive parameter counts demand prohibitive amounts of memory and computational power. This creates a massive bottleneck for practical deployment, especially in resource-constrained edge devices or high-concurrency cloud environments.

Quantization—reducing the bit-width of weights, activations, and gradients—offers a solution. However, LLMs present unique challenges that break traditional quantization methods. Most notably, LLMs exhibit emergent, high-magnitude "outlier" features in their activations and weights. Standard quantization forces these outliers and normal values into the same narrow bit-range, destroying model accuracy. This paper provides a structured taxonomy to help engineers and researchers navigate the complex landscape of algorithms and systems designed to tame these outliers and compress LLMs efficiently.

Methods

The survey organizes the low-bit LLM literature into three core pillars: Basics, Systems, and Algorithms.

Figure 1: The skeleton of the LLM Quantization methods. The diagram illustrates the main areas in the survey.
  • Basics (Data Formats & Granularity): The authors explore standard formats (INT8, INT4, FP8) alongside custom formats tailored specifically for LLMs, such as NormalFloat (NF4) and Student Float (SF4), which better fit the statistical distributions of model parameters. They also define quantization granularities ranging from coarse tensor-wise to fine-grained token-wise and group-wise strategies.
  • Systems and Frameworks: The paper reviews how mainstream inference engines (e.g., vLLM, TensorRT-LLM, llama.cpp) handle memory hierarchy bottlenecks. It contrasts weight-only quantization (compressing storage to alleviate memory bandwidth limits) with weight-activation and KV cache quantization.
  • Algorithmic Strategies: Algorithms are split into Quantization-Aware Training (QAT)—including extreme 1-bit architectures like BitNet—and Post-Training Quantization (PTQ). PTQ methods are further categorized into equivalent transformations (e.g., SmoothQuant, QuaRot), error compensation (e.g., GPTQ), and mixed-precision techniques (e.g., SpQR).

Datasets

The paper surveys the evaluation ecosystems used to validate quantized models. Rather than introducing a new dataset, it outlines how toolkits (such as LLMC, LMQuant, and QLLM-Eval) leverage established benchmarks:

  • Perplexity & Language Modeling: Evaluated using datasets like WikiText-2.
  • Zero-Shot Reasoning: Measured via the lm-evaluation-harness.
  • Advanced Capabilities: Evaluated using comprehensive frameworks like OpenCompass, LongEval (for long-context retention), and MT-Bench (for instruction-following and dialogue quality).

Key Insights and Ideas

  • The Outlier Migration Trick: Instead of trying to quantize severe activation outliers directly, pioneering methods like SmoothQuant use mathematically equivalent transformations to scale down activation outliers while scaling up the weights, effectively migrating the quantization difficulty to the more stable weight tensors.
  • Smearing Out Outliers with Rotations: Advanced PTQ methods like QuaRot and SpinQuant introduce orthogonal rotation matrices into the model's forward pass. This structurally "smears out" localized outliers across channels, making the entire tensor much easier to quantize uniformly.
  • Memory-Bound vs. Compute-Bound: During the autoregressive decode stage of LLM inference, the primary bottleneck is memory bandwidth (fetching weights from high-bandwidth memory to the GPU cache). Because of this, weight-only 4-bit quantization delivers massive real-world speedups, even though the weights must be dequantized back to FP16 on-the-fly to perform the actual math.

Key Results

The survey synthesizes quantitative milestones achieved across the literature:

  • Near-Lossless 4-bit PTQ: Methods like AWQ and GPTQ successfully compress LLM weights to 4 bits, achieving a ~4x reduction in memory footprint with negligible drops in accuracy across standard zero-shot tasks.
  • Throughput Gains via KV Cache Quantization: Compressing the Key-Value cache down to 4-bit or 2-bit (using frameworks like KIVI) dramatically reduces runtime memory consumption, allowing systems to double or quadruple maximum serving batch sizes and handle significantly longer contexts.
  • The Rise of 1-bit LLMs: Native 1-bit architectures (like BitNet b1.58) demonstrate that models trained from scratch using ternary weights (-1, 0, 1) can match the performance of full-precision models while offering orders-of-magnitude improvements in energy efficiency and compute throughput.

Limitations and Future Work

The authors highlight several open challenges guiding the next generation of quantization research:

  1. Mysterious Outlier Origins: The exact internal mechanisms causing low-semantic-value activation outliers remain poorly understood. Uncovering why they happen is critical to designing cleaner quantization boundaries.
  2. The 3-Bit Wall for PTQ: Pushing post-training quantization below 3 bits without severe accuracy degradation remains an unresolved hurdle.
  3. Inter-Layer Mixed Precision: While intra-layer mixed precision (treating outliers differently within a layer) is highly mature, unified strategies that dynamically allocate different bit-widths across different layers (inter-layer) are still under-explored.
  4. Architectural Frontiers: Extending low-bit quantization to complex, emerging architectures like Mixture-of-Experts (MoE) and multi-modal vision-language models introduces unique expert-load balancing and cross-modality challenges that require dedicated hardware-algorithm co-design.

Summary updated

Abstract

Large language models (LLMs) have achieved remarkable advancements in natural language processing, showcasing exceptional performance across various tasks. However, the expensive memory and computational requirements present significant challenges for their practical deployment. Low-bit quantization has emerged as a critical approach to mitigate these challenges by reducing the bit-width of model parameters, activations, and gradients, thus decreasing memory usage and computational demands. This paper presents a comprehensive survey of low-bit quantization methods tailored for LLMs, covering the fundamental principles, system implementations, and algorithmic strategies. An overview of basic concepts and new data formats specific to low-bit LLMs is first introduced, followed by a review of frameworks and systems that facilitate low-bit LLMs across various hardware platforms. Then, we categorize and analyze techniques and toolkits for efficient low-bit training and inference of LLMs. Finally, we conclude with a discussion of future trends and potential advancements of low-bit LLMs. Our systematic overview from basic, system, and algorithm perspectives can offer valuable insights and guidelines for future works to enhance the efficiency and applicability of LLMs through low-bit quantization.
BibTeX
@inproceedings{gong2024survey,
  author = {Ruihao Gong and Yifu Ding and Zining Wang and Chengtao Lv and Xingyu Zheng and Jinyang Du and Haotong Qin and Jinyang Guo and Michele Magno and Xianglong Liu},
  title = {{A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms}},
  booktitle = {Neural Networks},
  year = {2024},
  url = {https://openreview.net/forum?id=7hDdP6onVi},
}
Figure