TL;DR
Activation quantization in Large Language Models (LLMs) has long relied on costly, per-token dynamic quantization to cope with massive "token-wise outliers." PrefixQuant eliminates these extreme activation spikes offline without retraining by pre-populating the KV cache with high-frequency outlier tokens (attention sinks). By neutralizing token-wise variance, PrefixQuant allows hardware-friendly per-tensor static quantization to surpass dynamic quantization in both accuracy and inference throughput.
What Problem Does This Paper Address?
Quantizing both weights and activations to 4-bit (W4A4) drastically reduces memory usage and speeds up matrix multiplication. However, LLM activations suffer from extreme outliers:
- Channel-wise outliers: Specific hidden dimensions consistently exhibit huge values across tokens. Methods like Hadamard rotation (e.g., QuaRot) largely distribute these across channels.
- Token-wise outliers: Specific tokens (such as punctuation or sequence beginnings) exhibit values hundreds to thousands of times larger than normal tokens.
Because token magnitudes vary wildly across a sequence, static quantization (using one fixed scaling factor for the entire tensor) causes massive clipping or rounding errors. Practitioners have had to use per-token dynamic quantization, computing min/max scales on-the-fly during inference. Dynamic quantization introduces significant runtime latency (roughly $3\times$ slower quantization overhead) and complicates operator fusion in modern hardware kernels.
Methods
PrefixQuant resolves token-wise outliers by tackling their root cause: the self-attention mechanism’s tendency to dump surplus attention onto "sink" tokens.
-
Identifying and Profiling Outlier Tokens: The authors observe that token-wise outliers are identical across linear layers and self-attention projections ($Q/K/V$). Outliers are detected offline on a small calibration set using the inputs to the
down_projlayer by finding tokens whose maximum activation exceeds the sequence median by a factor $\eta$ (empirically $\eta = 64$). -
Prefixed Outliers in the KV Cache: Outlier tokens almost exclusively consist of the initial token (
[BOS]) or low-semantic delimiter tokens (.,\n,the). PrefixQuant identifies the top-$N$ most frequent outlier tokens (typically 1 to 4 tokens) and pre-computes their keys and values, placing them permanently in the KV cache prefix. During inference, attention heads naturally dump surplus attention into these pre-loaded sink tokens, preventing downstream tokens from becoming outliers. This process takes only 12 to 60 seconds offline. -
Hadamard Rotation & Static Calibration: PrefixQuant incorporates standard Hadamard rotations to smooth out channel-wise outliers, then runs a grid search on calibration data to determine optimal per-tensor static clipping thresholds ($s_X$).
-
Optional Block-Wise Fine-Tuning: Because static scale factors are fixed and differentiable, PrefixQuant optionally performs block-by-block reconstruction (optimizing mean squared error) across full-precision weights and quantization step sizes.
Datasets
- Calibration & Training: The Pile (default; 8 samples for grid search, 512 samples of length 1024 for fine-tuning). Ablations also evaluate C4 and RedPajama.
- Evaluation:
- Perplexity: WikiText-2 (context length 2048).
- Zero-Shot Reasoning: PIQA, ARC-Easy, ARC-Challenge, HellaSwag, and WinoGrande (evaluated via
lm-evalv0.4.2). - No new benchmarks were introduced.
Key Insights and Ideas
- Outliers Can Be Trapped Offline: Rather than altering the model architecture or retraining with modified attention (such as zero-SoftMax), seeding the KV cache with dummy sinks absorbs superfluous attention weights and flattens the dynamic range (reducing the peak-to-median ratio from >400 to <3).
- Static Can Beat Dynamic in Low-Bit Regimes: In 8-bit quantization, dynamic min/max scaling works well without clipping. But at 4-bit, balancing rounding error and clipping error is paramount. Calibrated static quantization with fixed, optimized clipping step sizes actually outperforms standard dynamic min/max scaling once the outlier tokens are removed.
- Fast Heuristic vs. Search: While earlier sink-prefixing attempts (e.g., CushionCache) required up to 12 hours of greedy search, frequency-based profiling of delimiter tokens achieves superior results in under a minute.
Key Results
- Accuracy at W4A4KV4:
- On Llama-3-8B, PrefixQuant with static quantization achieves a WikiText-2 perplexity of 7.43 and an average reasoning accuracy of 71.08%, substantially outperforming the dynamic baseline QuaRot (8.41 perplexity, 65.15% accuracy).
- On Llama-3-70B, PrefixQuant reaches 4.41 perplexity (vs. 6.82 for QuaRot).
- Inference Speed:
- The per-tensor static quantization kernel is $2.82\times$ to $3.31\times$ faster than per-token dynamic quantization kernels.
- In end-to-end W4A4 prefilling on Llama-3-8B, PrefixQuant delivers a $1.60\times$ to $2.81\times$ speedup over FP16 baselines and beats QuaRot by $1.2\times$ to $1.3\times$ across RTX 3090 and A100 GPUs.
- Weight-Only Quantization Plug-in: By preventing extreme activations from skewing MSE loss during block-wise reconstruction, PrefixQuant improves W2A16g128 EfficientQAT performance on Llama-3-8B by +5.05 points in average zero-shot accuracy.
Limitations and Future Work
- End-to-End Fine-Tuning Not Explored: The method currently uses sequential block-wise optimization; full end-to-end quantization-aware training (E2E-QAT) is left for future work.
- KV-Cache Overhead: Storing prefixed tokens incurs a tiny KV cache memory footprint (1–4 tokens), which is negligible for standard context lengths but fundamentally alters sequence indexing.
- Scope of Latency Benchmarks: Latency gains were primarily evaluated for computation-bound GEMM/GEMV prefilling stages; decoding-phase speedups under different batching regimes with quantized KV cache were not deeply explored in the hardware benchmarks.
Summary updated