Introduction
Quantizing Large Language Models (LLMs) is essential for deploying these massive architectures on consumer-grade hardware. While weight quantization has matured, quantizing activations remains a significant challenge due to "outliers"—extreme values in specific channels that disproportionately affect numerical precision. Current state-of-the-art methods like Atom and QuaRot address these outliers locally but often overlook how quantization errors propagate through the model.
This paper introduces BiSup, a framework that treats quantization error as a "butterfly effect" propagating in two directions: vertically through the model's layers and horizontally across the sequence of tokens. By suppressing errors in both dimensions, BiSup achieves significant performance gains in ultra-low-bit regimes.
The Bidirectional Error Problem
BiSup identifies two primary modes of error propagation:
- Vertical Accumulation: Quantization errors introduced in early layers are passed to subsequent layers. Each layer adds its own noise, eventually burying the original signal by the time it reaches the final output.
- Horizontal Diffusion: Due to the self-attention mechanism, tokens are interdependent. The authors observed that the "system prompt" (the initial tokens) acts as an anchor. If these tokens are poorly quantized, the error diffuses into every subsequent token generated by the model.
Methodology
Vertical Suppression: PEFT-based Optimization
To mitigate vertical error accumulation, BiSup utilizes a small amount of calibration data (e.g., 128 samples from WikiText2) to fine-tune quantization parameters using Parameter-Efficient Fine-Tuning (PEFT). This involves three core techniques:
- Fine-Grained Clipping: Instead of a single clipping value per tensor, BiSup employs learnable, per-group clipping values.
- Soft-Constrained Smoothing: Evolving the SmoothQuant approach, BiSup uses two learnable scaling factors ($s_1, s_2$) to find an optimal balance between weight and activation precision dynamically.
- Stabilized Low-Rank Error Compensation: A LoRA-like structure is used to compensate for errors, anchored to the original weights ($W \cdot (1 + AB)$) to maintain stability during short-duration fine-tuning.
Horizontal Suppression: Prompt Mixed-Precision
BiSup addresses horizontal diffusion by protecting the most critical tokens in the sequence. The authors found that maintaining the Key-Value (KV) cache for system prompts in high precision (FP16/BF16) while quantizing the rest of the sequence significantly stabilizes the model.
As shown in the figure above, keeping the System Prompt (SP) in floating-point (FP) precision while allowing the User Prompt (UP) and New Tokens (NT) to be processed in integer (INT) precision prevents the initial quantization noise from corrupting the entire generation process.
Experimental Results
BiSup was evaluated across Llama and Qwen model families. The most dramatic improvements were observed in the ultra-low-bit W3A3 (3-bit weights, 3-bit activations) setting, which is typically considered the "danger zone" for LLM stability.
On Llama 3-8B (W3A3-g128), BiSup reduced WikiText2 perplexity from 13.26 (Atom) and 14.33 (QuaRot) to 7.85. Furthermore, zero-shot accuracy across six benchmarks improved from an average of 46.93% to 55.67%, demonstrating that the method effectively preserves the model's reasoning capabilities even under extreme compression.
Conclusion
BiSup demonstrates that addressing the propagation of quantization errors is as important as addressing the errors themselves. By combining learnable vertical suppression with a high-precision anchor for horizontal sequences, BiSup enables stable LLM deployment at bit-widths previously thought to be unusable for general-purpose tasks.
Summary updated