PrefixQuant: Static Quantization Beats Dynamic through Prefixed Outliers in LLMs

Mengzhao Chen, Yi Liu, Jiahao Wang, Yi Bin, Wenqi Shao, Ping LuoSubmission to ICLR 2025 · Rejected

TL;DR

Activation quantization in Large Language Models (LLMs) has long relied on costly, per-token dynamic quantization to cope with massive "token-wise outliers." PrefixQuant eliminates these extreme activation spikes offline without retraining by pre-populating the KV cache with high-frequency outlier tokens (attention sinks). By neutralizing token-wise variance, PrefixQuant allows hardware-friendly per-tensor static quantization to surpass dynamic quantization in both accuracy and inference throughput.

What Problem Does This Paper Address?

Quantizing both weights and activations to 4-bit (W4A4) drastically reduces memory usage and speeds up matrix multiplication. However, LLM activations suffer from extreme outliers:

  1. Channel-wise outliers: Specific hidden dimensions consistently exhibit huge values across tokens. Methods like Hadamard rotation (e.g., QuaRot) largely distribute these across channels.
  2. Token-wise outliers: Specific tokens (such as punctuation or sequence beginnings) exhibit values hundreds to thousands of times larger than normal tokens.

Because token magnitudes vary wildly across a sequence, static quantization (using one fixed scaling factor for the entire tensor) causes massive clipping or rounding errors. Practitioners have had to use per-token dynamic quantization, computing min/max scales on-the-fly during inference. Dynamic quantization introduces significant runtime latency (roughly $3\times$ slower quantization overhead) and complicates operator fusion in modern hardware kernels.

Methods

PrefixQuant resolves token-wise outliers by tackling their root cause: the self-attention mechanism’s tendency to dump surplus attention onto "sink" tokens.

  • Identifying and Profiling Outlier Tokens: The authors observe that token-wise outliers are identical across linear layers and self-attention projections ($Q/K/V$). Outliers are detected offline on a small calibration set using the inputs to the down_proj layer by finding tokens whose maximum activation exceeds the sequence median by a factor $\eta$ (empirically $\eta = 64$).

  • Prefixed Outliers in the KV Cache: Outlier tokens almost exclusively consist of the initial token ([BOS]) or low-semantic delimiter tokens (., \n, the). PrefixQuant identifies the top-$N$ most frequent outlier tokens (typically 1 to 4 tokens) and pre-computes their keys and values, placing them permanently in the KV cache prefix. During inference, attention heads naturally dump surplus attention into these pre-loaded sink tokens, preventing downstream tokens from becoming outliers. This process takes only 12 to 60 seconds offline.

  • Hadamard Rotation & Static Calibration: PrefixQuant incorporates standard Hadamard rotations to smooth out channel-wise outliers, then runs a grid search on calibration data to determine optimal per-tensor static clipping thresholds ($s_X$).

  • Optional Block-Wise Fine-Tuning: Because static scale factors are fixed and differentiable, PrefixQuant optionally performs block-by-block reconstruction (optimizing mean squared error) across full-precision weights and quantization step sizes.

Datasets

  • Calibration & Training: The Pile (default; 8 samples for grid search, 512 samples of length 1024 for fine-tuning). Ablations also evaluate C4 and RedPajama.
  • Evaluation:
  • Perplexity: WikiText-2 (context length 2048).
  • Zero-Shot Reasoning: PIQA, ARC-Easy, ARC-Challenge, HellaSwag, and WinoGrande (evaluated via lm-eval v0.4.2).
  • No new benchmarks were introduced.

Key Insights and Ideas

  • Outliers Can Be Trapped Offline: Rather than altering the model architecture or retraining with modified attention (such as zero-SoftMax), seeding the KV cache with dummy sinks absorbs superfluous attention weights and flattens the dynamic range (reducing the peak-to-median ratio from >400 to <3).
  • Static Can Beat Dynamic in Low-Bit Regimes: In 8-bit quantization, dynamic min/max scaling works well without clipping. But at 4-bit, balancing rounding error and clipping error is paramount. Calibrated static quantization with fixed, optimized clipping step sizes actually outperforms standard dynamic min/max scaling once the outlier tokens are removed.
  • Fast Heuristic vs. Search: While earlier sink-prefixing attempts (e.g., CushionCache) required up to 12 hours of greedy search, frequency-based profiling of delimiter tokens achieves superior results in under a minute.

Key Results

  • Accuracy at W4A4KV4:
  • On Llama-3-8B, PrefixQuant with static quantization achieves a WikiText-2 perplexity of 7.43 and an average reasoning accuracy of 71.08%, substantially outperforming the dynamic baseline QuaRot (8.41 perplexity, 65.15% accuracy).
  • On Llama-3-70B, PrefixQuant reaches 4.41 perplexity (vs. 6.82 for QuaRot).
  • Inference Speed:
  • The per-tensor static quantization kernel is $2.82\times$ to $3.31\times$ faster than per-token dynamic quantization kernels.
  • In end-to-end W4A4 prefilling on Llama-3-8B, PrefixQuant delivers a $1.60\times$ to $2.81\times$ speedup over FP16 baselines and beats QuaRot by $1.2\times$ to $1.3\times$ across RTX 3090 and A100 GPUs.
  • Weight-Only Quantization Plug-in: By preventing extreme activations from skewing MSE loss during block-wise reconstruction, PrefixQuant improves W2A16g128 EfficientQAT performance on Llama-3-8B by +5.05 points in average zero-shot accuracy.

Limitations and Future Work

  • End-to-End Fine-Tuning Not Explored: The method currently uses sequential block-wise optimization; full end-to-end quantization-aware training (E2E-QAT) is left for future work.
  • KV-Cache Overhead: Storing prefixed tokens incurs a tiny KV cache memory footprint (1–4 tokens), which is negligible for standard context lengths but fundamentally alters sequence indexing.
  • Scope of Latency Benchmarks: Latency gains were primarily evaluated for computation-bound GEMM/GEMV prefilling stages; decoding-phase speedups under different batching regimes with quantized KV cache were not deeply explored in the hardware benchmarks.

Summary updated

Abstract

Quantization is essential for deploying Large Language Models (LLMs) by enhancing memory efficiency and inference speed. Existing methods for activation quantization mainly address channel-wise outliers, often neglecting token-wise outliers, leading to reliance on costly per-token dynamic quantization. To address this, we introduce PrefixQuant, a novel technique that isolates outlier tokens offline without re-training. Specifically, PrefixQuant identifies high-frequency outlier tokens and prefixes them in the KV cache, preventing the generation of outlier tokens during inference and simplifying quantization. To our knowledge, PrefixQuant is the first to enable efficient per-tensor static quantization to outperform expensive per-token dynamic quantization. For instance, in W4A4KV4 (4- bit weight, 4-bit activation, and 4-bit KV cache) Llama-3-8B, PrefixQuant with per-tensor static quantization achieves a 7.43 WikiText2 perplexity and 71.08% average accuracy on 5 common-sense reasoning tasks, outperforming previous per-token dynamic quantization methods like QuaRot with 0.98 perplexity improvement and +5.98 points accuracy. Additionally, the inference speed of W4A4 quantized models using PrefixQuant is 1.60× to 2.81× faster than FP16 models and exceeds QuaRot models by 1.2× to 1.3×.
BibTeX
@misc{chen2024prefixquant,
  author = {Mengzhao Chen and Yi Liu and Jiahao Wang and Yi Bin and Wenqi Shao and Ping Luo},
  title = {{PrefixQuant: Static Quantization Beats Dynamic through Prefixed Outliers in LLMs}},
  year = {2024},
  eprint = {2410.05265},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.05265},
}

Loading related content...

Figure