PrefixQuant: Static Quantization Beats Dynamic through Prefixed Outliers in LLMs

Mengzhao Chen, Yi Liu, Jiahao Wang, Yi Bin, Wenqi Shao, Ping Luo

Abstract

Existing weight-activation quantization methods for Large Language Models (LLMs) primarily address channel-wise outliers but often neglect token-wise outliers, which limits the accuracy of quantized models. In this work, we propose PrefixQuant, a novel quantization method that achieves state-of-the-art performance across various precision levels (W4A4KV4 and W4A8KV4) and granularities (dynamic and static quantization) by effectively isolating token-wise outliers. First, PrefixQuant eliminates token-wise outliers by prefixing outlier tokens in the KV cache, a process that is training-free and highly efficient. Second, PrefixQuant introduces new trainable parameters for block-wise training to compensate for quantization error. Our experiments show that PrefixQuant significantly outperforms existing dynamic quantization methods, even under coarser static quantization settings. For instance, PrefixQuant achieves an average accuracy improvement of +3.08 and +2.85 points over SpinQuant (dynamic quantization) on five zero-shot reasoning tasks under dynamic and static quantization settings, respectively, on W4A4KV4 Llama-3-8B. Additionally, we demonstrate up to 2.74x prefilling speedup and 2.16x decoding speedup for LLMs using W4A4 PrefixQuant.

Introduction

Quantizing Large Language Models (LLMs)—reducing the precision of weights and activations to 4 or 8 bits—is essential for deploying these massive models on consumer hardware. However, quantization often leads to significant accuracy drops due to outliers: activation values that are orders of magnitude larger than the average. While existing research has focused heavily on "channel-wise" outliers, this paper highlights a neglected culprit: token-wise outliers.

The Token-Wise Outlier Challenge

Token-wise outliers are activations that are significantly larger than the mean, occurring at specific token positions. These "massive activations" make it nearly impossible to find a single scaling factor that works for an entire sequence without losing critical information. As shown in Figure 1, in a sequence of 2048 tokens, just two outlier tokens can account for a staggering 94.7% of the total quantization error.

4-bit per-token dynamic quantization error in 2048 input context length. Two outlier tokens account for 94.7% of quantization error.

PrefixQuant Methodology

The core innovation of PrefixQuant is a two-step process to isolate and compensate for these outliers:

1. Outlier Isolation via Prefixed Tokens

The authors observed that token-wise outliers are not random; they consistently appear at the start of a sequence (the "attention sink" phenomenon) or are triggered by specific high-frequency tokens like periods (.), newlines (\n), or the [BOS] token. PrefixQuant identifies these "problematic" tokens for a given model and prefixes them to every input. These prefixed tokens are pre-processed in full precision, and their Key-Value (KV) pairs are stored in a permanent, full-precision cache. During inference, the outliers are "trapped" within this prefixed cache, leaving the rest of the user's input with a much smoother, quantization-friendly distribution.

2. Block-wise Fine-tuning

To further bridge the accuracy gap, the authors introduce a training-efficient step. They optimize the model's weights and quantization parameters (like clipping factors) block-by-block using a small calibration dataset. This helps the model adapt to the slight noise introduced by the low-bit quantization of the remaining tokens.

Comparison of proposed PrefixQuant with existing methods. Prefixing specific tokens in the KV cache isolates outliers and significantly reduces quantization error.

Experimental Evaluation

The researchers evaluated PrefixQuant across a wide range of popular LLM families, including Llama-2, Llama-3, Mistral-7B, and Qwen-2.

  • Accuracy Gains: On W4A4KV4 Llama-3-8B, PrefixQuant improved average zero-shot accuracy by +3.08 points over the previous leader, SpinQuant. Remarkably, PrefixQuant's static quantization version outperformed the dynamic versions of previous state-of-the-art methods.
  • Hardware Speedup: By enabling efficient 4-bit integer arithmetic, the method achieved a 2.74× speedup in prefilling and a 2.16× speedup in decoding compared to standard FP16 models on an NVIDIA RTX 3090.
  • Efficiency: The process of identifying outlier tokens is incredibly fast, taking only about 1 minute for a Llama-3-70B model.

Conclusion

PrefixQuant demonstrates that token-wise outliers are a primary bottleneck in LLM quantization. By isolating these outliers in a prefixed, full-precision KV cache, the method allows static quantization to achieve superior accuracy and significant inference speedups. This approach provides a practical path for deploying high-performance LLMs on resource-constrained hardware.

Summary updated

Abstract

Existing weight-activation quantization methods for Large Language Models (LLMs) primarily address channel-wise outliers but often neglect token-wise outliers, which limits the accuracy of quantized models. In this work, we propose PrefixQuant, a novel quantization method that achieves state-of-the-art performance across various precision levels (W4A4KV4 and W4A8KV4) and granularities (dynamic and static quantization) by effectively isolating token-wise outliers. First, PrefixQuant eliminates token-wise outliers by prefixing outlier tokens in the KV cache, a process that is training-free and highly efficient (e.g., 1 minutes for Llama-3-70B). Second, PrefixQuant introduces new trainable parameters for block-wise training to compensate for quantization error. Our experiments show that PrefixQuant significantly outperforms existing dynamic quantization methods, even under coarser static quantization settings. For instance, PrefixQuant achieves an average accuracy improvement of +3.08 and +2.85 points over SpinQuant (dynamic quantization) on five zero-shot reasoning tasks under dynamic and static quantization settings, respectively, on W4A4KV4 Llama-3-8B. Additionally, we demonstrate up to 2.74x prefilling speedup and 2.16x decoding speedup for LLMs using W4A4 PrefixQuant. Our code is available at https://github.com/ChenMnZ/PrefixQuant.
BibTeX
@misc{chen2024prefixquant,
  author = {Mengzhao Chen and Yi Liu and Jiahao Wang and Yi Bin and Wenqi Shao and Ping Luo},
  title = {{PrefixQuant: Static Quantization Beats Dynamic through Prefixed Outliers in LLMs}},
  year = {2024},
  eprint = {2410.05265},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.05265},
}
Figure