Abstract
As Large Language Models (LLMs) continue to scale, the memory requirements for deployment have become a significant bottleneck for consumer-grade hardware. While 4-bit quantization has emerged as a standard for near-lossless compression, moving to extreme bit-widths (2 or 3 bits) typically results in a "perplexity cliff"—a sharp degradation in model performance. This paper introduces Additive Quantization of Language Models (AQLM), a framework that leverages Multi-Codebook Quantization (MCQ) and block-level optimization to achieve state-of-the-art accuracy in the sub-3-bit regime. AQLM enables models like Llama 2 70B to run on single consumer GPUs with minimal performance loss, while providing inference speeds that rival or exceed FP16 implementations.
Introduction
The deployment of LLMs is fundamentally constrained by GPU VRAM. While 4-bit quantization methods like GPTQ or SpQR have become popular, they often fail at 2-bit widths because they rely on rounding weights to a simple grid. Even advanced geometric approaches like QuIP# struggle to maintain intelligence at these extreme levels.
AQLM addresses this by revisiting Multi-Codebook Quantization (MCQ), a technique traditionally used in information retrieval. By representing weight groups as sums of learned vectors rather than individual scalar points, AQLM provides a more expressive representation for a given memory footprint. This allows the model to reach a new "Pareto frontier," where a larger model compressed to 2 bits can actually outperform a smaller model at higher precision.
Methodology
AQLM is a data-aware quantization scheme that optimizes weights based on their impact on the model's output. The process is divided into three distinct phases:
1. Multi-Codebook Representation
Instead of quantizing each weight individually, AQLM groups weights into vectors. Each vector is approximated as a sum of $M$ codewords, where each codeword is selected from a separate learned codebook. This additive approach allows for a much higher degree of nuance in the weight distribution compared to standard scalar quantization.
2. Optimization Pipeline
- Beam Search for Codes: For a specific weight matrix and a set of calibration data (e.g., from RedPajama), the algorithm employs a beam search to find the optimal combination of discrete codewords that minimizes the reconstruction error of the layer's output.
- Codebook Update: After fixing the discrete codes, the vectors within the codebooks are optimized using the Adam optimizer. This step further reduces the squared error between the original and quantized layer outputs.
- Intra-block Fine-tuning: To mitigate the compounding of quantization errors across layers, AQLM performs joint optimization across entire transformer blocks (typically 4-8 layers). This ensures that the parameters (both codebooks and scales) are tuned to maintain the integrity of the model's internal representations.
3. Hardware-Efficient Kernels
To ensure practical utility, the authors developed high-performance CUDA and Triton kernels. These kernels are designed to handle the complexity of MCQ efficiently, enabling real-time token generation that is often faster than standard FP16 execution.
Experimental Setup
The framework was evaluated using the Llama 2 (7B, 13B, 70B) and Mixtral-8x7B model families.
- Calibration: 8 million tokens from the RedPajama-v1 dataset.
- Metrics: Perplexity was measured on WikiText-2 and C4. Zero-shot accuracy was tested across a diverse suite of benchmarks, including MMLU, GSM8k, HellaSwag, and ARC.
Key Results and Insights
Breaking the Perplexity Cliff
AQLM demonstrates a significant leap in accuracy at low bit-rates. For the Llama 2 70B model, 2-bit AQLM achieved a WikiText-2 perplexity of 3.94, outperforming the previous state-of-the-art, QuIP#, which scored 4.16. In zero-shot tasks, 2-bit AQLM models frequently retained 90-95% of the original 16-bit model's performance.
The 2-Bit Pareto Frontier
A critical finding of this research is that AQLM makes extreme compression a viable strategy for model selection. Previously, a 7B model at 4 bits almost always outperformed a 13B model at 2 bits. With AQLM, this trend is reversed: the 13B 2-bit model now provides better accuracy for the same memory footprint as the 7B 4-bit model.
Inference Speed
Despite the mathematical complexity of additive quantization, the optimized kernels provide a 1.2x to 3x speedup over FP16 on an NVIDIA RTX 3090. This proves that AQLM is not just a theoretical improvement but a practical solution for high-throughput inference.
Limitations and Future Work
The primary trade-off in AQLM is the "offline" quantization cost. Quantizing a 70B parameter model can take up to two weeks on a single A100 GPU, though this can be reduced to approximately three days through parallelization across 8 GPUs. Additionally, the method is sensitive to hyperparameters like group size and the number of codebooks, which may require tuning for new architectures.
Future research directions include applying AQLM to KV cache compression for long-context windows, extending the method to vision transformers, and integrating end-to-end distillation to further close the gap with full-precision models.
Conclusion
AQLM represents a significant milestone in LLM compression. By shifting from scalar rounding to learned additive vector quantization, it enables the deployment of massive models on consumer hardware without the catastrophic loss of intelligence previously associated with the 2-bit regime.
Summary updated