BiSup: Bidirectional Quantization Error Suppression for Large Language Models

Zhiyong Feng, Minghui Zou, Ronghui Guo, Sai Zhang, Xiaowang Zhang, Zhiyong Feng 0002arXiv 2024 · DOI · arXiv

Introduction

Quantizing Large Language Models (LLMs) is essential for deploying these massive architectures on consumer-grade hardware. While weight quantization has matured, quantizing activations remains a significant challenge due to "outliers"—extreme values in specific channels that disproportionately affect numerical precision. Current state-of-the-art methods like Atom and QuaRot address these outliers locally but often overlook how quantization errors propagate through the model.

This paper introduces BiSup, a framework that treats quantization error as a "butterfly effect" propagating in two directions: vertically through the model's layers and horizontally across the sequence of tokens. By suppressing errors in both dimensions, BiSup achieves significant performance gains in ultra-low-bit regimes.

The Bidirectional Error Problem

BiSup identifies two primary modes of error propagation:

  1. Vertical Accumulation: Quantization errors introduced in early layers are passed to subsequent layers. Each layer adds its own noise, eventually burying the original signal by the time it reaches the final output.
  2. Horizontal Diffusion: Due to the self-attention mechanism, tokens are interdependent. The authors observed that the "system prompt" (the initial tokens) acts as an anchor. If these tokens are poorly quantized, the error diffuses into every subsequent token generated by the model.

Methodology

Vertical Suppression: PEFT-based Optimization

To mitigate vertical error accumulation, BiSup utilizes a small amount of calibration data (e.g., 128 samples from WikiText2) to fine-tune quantization parameters using Parameter-Efficient Fine-Tuning (PEFT). This involves three core techniques:

  • Fine-Grained Clipping: Instead of a single clipping value per tensor, BiSup employs learnable, per-group clipping values.
  • Soft-Constrained Smoothing: Evolving the SmoothQuant approach, BiSup uses two learnable scaling factors ($s_1, s_2$) to find an optimal balance between weight and activation precision dynamically.
  • Stabilized Low-Rank Error Compensation: A LoRA-like structure is used to compensate for errors, anchored to the original weights ($W \cdot (1 + AB)$) to maintain stability during short-duration fine-tuning.

Horizontal Suppression: Prompt Mixed-Precision

BiSup addresses horizontal diffusion by protecting the most critical tokens in the sequence. The authors found that maintaining the Key-Value (KV) cache for system prompts in high precision (FP16/BF16) while quantizing the rest of the sequence significantly stabilizes the model.

Figure 5: (a, b) Attention heatmaps showing the importance of initial tokens; (c) The proposed mixed-precision pipeline for horizontal error suppression.

As shown in the figure above, keeping the System Prompt (SP) in floating-point (FP) precision while allowing the User Prompt (UP) and New Tokens (NT) to be processed in integer (INT) precision prevents the initial quantization noise from corrupting the entire generation process.

Experimental Results

BiSup was evaluated across Llama and Qwen model families. The most dramatic improvements were observed in the ultra-low-bit W3A3 (3-bit weights, 3-bit activations) setting, which is typically considered the "danger zone" for LLM stability.

Figure 6: Perplexity (PPL) results for Llama models. BiSup consistently outperforms Atom and QuaRot baselines, especially in W3A3 configurations.

On Llama 3-8B (W3A3-g128), BiSup reduced WikiText2 perplexity from 13.26 (Atom) and 14.33 (QuaRot) to 7.85. Furthermore, zero-shot accuracy across six benchmarks improved from an average of 46.93% to 55.67%, demonstrating that the method effectively preserves the model's reasoning capabilities even under extreme compression.

Conclusion

BiSup demonstrates that addressing the propagation of quantization errors is as important as addressing the errors themselves. By combining learnable vertical suppression with a high-precision anchor for horizontal sequences, BiSup enables stable LLM deployment at bit-widths previously thought to be unusable for general-purpose tasks.

Summary updated

Abstract

As the size and context length of Large Language Models (LLMs) grow, weight-activation quantization has emerged as a crucial technique for efficient deployment of LLMs. Compared to weight-only quantization, weight-activation quantization presents greater challenges due to the presence of outliers in activations. Existing methods have made significant progress by exploring mixed-precision quantization and outlier suppression. However, these methods primarily focus on optimizing the results of single matrix multiplication, neglecting the bidirectional propagation of quantization errors in LLMs. Specifically, errors accumulate vertically within the same token through layers, and diffuse horizontally across different tokens due to self-attention mechanisms. To address this issue, we introduce BiSup, a Bidirectional quantization error Suppression method. By constructing appropriate optimizable parameter spaces, BiSup utilizes a small amount of data for quantization-aware parameter-efficient fine-tuning to suppress the error vertical accumulation. Besides, BiSup employs prompt mixed-precision quantization strategy, which preserves high precision for the key-value cache of system prompts, to mitigate the error horizontal diffusion. Extensive experiments on Llama and Qwen families demonstrate that BiSup can improve performance over two state-of-the-art methods (the average WikiText2 perplexity decreases from 13.26 to 9.41 for Atom and from 14.33 to 7.85 for QuaRot under the W3A3-g128 configuration), further facilitating the practical applications of low-bit weight-activation quantization.
BibTeX
@misc{feng2024bisup,
  author = {Zhiyong Feng and Minghui Zou and Ronghui Guo and Sai Zhang and Xiaowang Zhang and Zhiyong Feng},
  title = {{BiSup: Bidirectional Quantization Error Suppression for Large Language Models}},
  year = {2024},
  eprint = {2405.15346},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.15346},
}
Figure