Abstract
Layer-wise quantization is a key technique for efficiently compressing large models without expensive retraining. Previous methods typically quantize the weights of each layer by "uniformly" optimizing the layer reconstruction loss across all output tokens. However, in this paper, we demonstrate that better-quantized models can be obtained by prioritizing learning from important tokens (e.g., those with large attention scores). Building on this finding, we propose RSQ (Rotate, Scale, then Quantize), which (1) applies rotations to mitigate outliers, (2) scales token features based on importance, and (3) quantizes the model using the GPTQ framework with scaled second-order statistics. We demonstrate that RSQ consistently outperforms baseline methods across multiple downstream tasks and model families, including LLaMA3, Mistral, and Qwen2.5, particularly in long-context scenarios.
Introduction
Large Language Models (LLMs) are computationally demanding, making quantization—reducing weight precision to 3 or 4 bits—essential for deployment. Most state-of-the-art post-training quantization (PTQ) methods, such as GPTQ and QuaRot, optimize layer reconstruction loss uniformly across all tokens in a calibration dataset. However, LLMs do not treat all tokens equally; "attention sinks" and initial tokens often carry disproportionate weight in determining model output. RSQ challenges the uniform optimization paradigm by prioritizing critical tokens during the quantization process.
The RSQ Framework
The authors propose a three-step framework to optimize quantization by focusing on token significance:
- Rotate: Building on the QuaRot method, RSQ applies orthogonal transformations (randomized Hadamard matrices) to the model's weights and activations. This "smooths out" massive outliers in the weight distribution, making them easier to quantize without losing precision.
- Scale: This is the core innovation. Instead of treating all tokens equally, RSQ scales input features based on token importance. While various strategies were explored, Attention Concentration (AttnCon) proved most effective. AttnCon calculates importance by summing the attention scores a token receives from all other tokens in a layer.
- Quantize: Finally, the GPTQ framework is used to quantize the weights. Crucially, the second-order statistics (the Hessian matrix) used by GPTQ are computed using these scaled tokens, forcing the optimizer to minimize reconstruction error more aggressively for important tokens.
To prevent the model from becoming overly biased toward specific positions, RSQ introduces Dataset Expansion. This involves shifting calibration sequences so that every token eventually occupies an "important" position during the quantization process.
Key Insights
"Less is More" in Calibration
A surprising finding of this research is that quantizing an LLM using only a subset of initial tokens can yield better performance than using the entire sequence. As shown in the perplexity analysis, using approximately the first 256 to 512 tokens (roughly 6-12% of a standard 4096-token sequence) often results in lower perplexity than using the full sequence. This confirms that initial tokens are vital for model stability and that uniform weighting can introduce noise from less relevant tokens.
Attention as an Importance Proxy
While heuristics like "First-N" tokens are effective, Attention Concentration is the superior metric for importance. Tokens that are heavily attended to are those the model relies on most for its internal representations; preserving their features during quantization is critical for maintaining performance.
Experimental Results
The authors evaluated RSQ on LLaMA3-8B, Mistral-Nemo-12B, and Qwen2.5-7B, primarily at 3-bit precision.
- Downstream Accuracy: RSQ consistently outperformed the QuaRot baseline, achieving absolute average accuracy improvements of 1.6% for LLaMA3, 0.9% for Mistral, and 0.4% for Qwen2.5 across ten downstream tasks.
- Long-Context Performance: On long-context benchmarks, RSQ showed even more pronounced gains, improving performance by up to 3.0% over QuaRot. This suggests that prioritizing important tokens helps the model maintain its reasoning chain over long sequences.
- Extreme Compression: The performance gap between RSQ and standard methods widens at lower bit-widths (e.g., 2-bit), proving that importance-aware scaling is a vital strategy for extreme model compression.
Conclusion
RSQ demonstrates that not all tokens are created equal in the eyes of a quantizer. By rotating to manage outliers and scaling to prioritize high-attention tokens, RSQ achieves state-of-the-art results for low-bit LLMs. Future work may involve exploring module-specific sensitivity to importance scaling and extending the framework to non-transformer architectures like Mixture-of-Experts (MoE) models.
Summary updated