Abstract
Existing weight-activation quantization methods for Large Language Models (LLMs) primarily address channel-wise outliers but often neglect token-wise outliers, which limits the accuracy of quantized models. In this work, we propose PrefixQuant, a novel quantization method that achieves state-of-the-art performance across various precision levels (W4A4KV4 and W4A8KV4) and granularities (dynamic and static quantization) by effectively isolating token-wise outliers. First, PrefixQuant eliminates token-wise outliers by prefixing outlier tokens in the KV cache, a process that is training-free and highly efficient. Second, PrefixQuant introduces new trainable parameters for block-wise training to compensate for quantization error. Our experiments show that PrefixQuant significantly outperforms existing dynamic quantization methods, even under coarser static quantization settings. For instance, PrefixQuant achieves an average accuracy improvement of +3.08 and +2.85 points over SpinQuant (dynamic quantization) on five zero-shot reasoning tasks under dynamic and static quantization settings, respectively, on W4A4KV4 Llama-3-8B. Additionally, we demonstrate up to 2.74x prefilling speedup and 2.16x decoding speedup for LLMs using W4A4 PrefixQuant.
Introduction
Quantizing Large Language Models (LLMs)—reducing the precision of weights and activations to 4 or 8 bits—is essential for deploying these massive models on consumer hardware. However, quantization often leads to significant accuracy drops due to outliers: activation values that are orders of magnitude larger than the average. While existing research has focused heavily on "channel-wise" outliers, this paper highlights a neglected culprit: token-wise outliers.
The Token-Wise Outlier Challenge
Token-wise outliers are activations that are significantly larger than the mean, occurring at specific token positions. These "massive activations" make it nearly impossible to find a single scaling factor that works for an entire sequence without losing critical information. As shown in Figure 1, in a sequence of 2048 tokens, just two outlier tokens can account for a staggering 94.7% of the total quantization error.
PrefixQuant Methodology
The core innovation of PrefixQuant is a two-step process to isolate and compensate for these outliers:
1. Outlier Isolation via Prefixed Tokens
The authors observed that token-wise outliers are not random; they consistently appear at the start of a sequence (the "attention sink" phenomenon) or are triggered by specific high-frequency tokens like periods (.), newlines (\n), or the [BOS] token. PrefixQuant identifies these "problematic" tokens for a given model and prefixes them to every input. These prefixed tokens are pre-processed in full precision, and their Key-Value (KV) pairs are stored in a permanent, full-precision cache. During inference, the outliers are "trapped" within this prefixed cache, leaving the rest of the user's input with a much smoother, quantization-friendly distribution.
2. Block-wise Fine-tuning
To further bridge the accuracy gap, the authors introduce a training-efficient step. They optimize the model's weights and quantization parameters (like clipping factors) block-by-block using a small calibration dataset. This helps the model adapt to the slight noise introduced by the low-bit quantization of the remaining tokens.
Experimental Evaluation
The researchers evaluated PrefixQuant across a wide range of popular LLM families, including Llama-2, Llama-3, Mistral-7B, and Qwen-2.
- Accuracy Gains: On W4A4KV4 Llama-3-8B, PrefixQuant improved average zero-shot accuracy by +3.08 points over the previous leader, SpinQuant. Remarkably, PrefixQuant's static quantization version outperformed the dynamic versions of previous state-of-the-art methods.
- Hardware Speedup: By enabling efficient 4-bit integer arithmetic, the method achieved a 2.74× speedup in prefilling and a 2.16× speedup in decoding compared to standard FP16 models on an NVIDIA RTX 3090.
- Efficiency: The process of identifying outlier tokens is incredibly fast, taking only about 1 minute for a Llama-3-70B model.
Conclusion
PrefixQuant demonstrates that token-wise outliers are a primary bottleneck in LLM quantization. By isolating these outliers in a prefixed, full-precision KV cache, the method allows static quantization to achieve superior accuracy and significant inference speedups. This approach provides a practical path for deploying high-performance LLMs on resource-constrained hardware.
Summary updated