TL;DR
This paper presents a comprehensive empirical study benchmarking 10 different quantization techniques on Meta's LLaMA3 models (8B and 70B) and the LLaMA3-based multimodal model LLaVA-Next-8B. The authors reveal that while 4-bit quantization delivers near-lossless performance, ultra-low bit-widths (≤3 bits) cause severe degradation—particularly in multimodal tasks—and expose a surprising limitation where standard LoRA fine-tuning actually worsens quantization errors.
What Problem Does This Paper Address?
Large Language Models (LLMs) like LLaMA3 achieve state-of-the-art performance but demand massive computational and memory resources, making deployment on edge devices or resource-constrained environments highly challenging. Quantization—reducing the bit-width of model weights and activations (e.g., from 16-bit floating-point to 4-bit or 2-bit integers)—is a primary strategy to compress these models.
However, existing quantization benchmarks have primarily focused on older architectures like LLaMA1 and LLaMA2. Because LLaMA3 was pre-trained on a massive scale of over 15 trillion tokens, its internal knowledge structure and sensitivity to quantization noise differ significantly. This paper addresses the critical gap of understanding how modern, heavily pre-trained LLMs and Multi-Modal LLMs (MLLMs) behave under various low-bit compression methods.
Methods
The authors systematically evaluate two main technical tracks across a spectrum of 1 to 8 bits:
- Post-Training Quantization (PTQ): They evaluate 9 prominent weight-only and weight-activation PTQ methods: RTN (Round-to-Nearest), GPTQ, AWQ, SmoothQuant, PB-LLM, QuIP, DB-LLM, BiLLM, and SliM-LLM.
- LoRA Fine-Tuning (LoRA-FT): They evaluate 2 popular low-rank adaptation methods designed for quantized models: QLoRA and IR-QLoRA.
These methods are applied to LLaMA3-8B and LLaMA3-70B for text tasks, and to the LLM backbone of LLaVA-Next-8B for vision-language tasks.
Datasets
The quantization methods are validated across a diverse suite of benchmarks:
- Language Perplexity (PPL): WikiText2, C4, and PTB.
- Zero-Shot CommonSenseQA: PIQA, Winogrande, ARC-e, ARC-c, and HellaSwag.
- Multi-Task Language Understanding: MMLU (specifically for evaluating LoRA fine-tuning).
- Multimodal Vision-Language Tasks: AI2D, ChartQA, DocVQA, MMBench, and MME (Cognition and Perception).
Key Insights and Ideas
- The LoRA Fine-Tuning Paradox: In a striking departure from LLaMA1 and LLaMA2 behavior, low-rank fine-tuning on the Alpaca dataset exacerbated performance degradation rather than fixing quantization errors. The authors hypothesize that because LLaMA3's base capabilities are so highly optimized by massive pre-training, fine-tuning on a small, generic dataset with limited parameters disrupts its sophisticated internal knowledge base.
- Information Redundancy in Larger Scales: LLaMA3-70B demonstrated remarkable robustness against quantization compared to the 8B variant. As bit-width decreased, the 70B model's performance degraded much more slowly, confirming that larger models possess significant information redundancy that buffers against quantization noise.
- The Multimodal Collapse: While 2-bit quantized models can still output coherent text sentences in pure language tasks, 2-bit quantization causes an absolute collapse in multimodal tasks (scores dropping to zero or generating repetitive gibberish). Visual-language alignment is exceptionally fragile and demands specialized mixed-precision handling (like SliM-LLM) to retain any utility at ultra-low bits.
Key Results
- 4-Bit Quantization is Safe: Across almost all PTQ methods, 4-bit quantization incurs a negligible performance drop (around 2% compared to the uncompressed FP16 baseline) while drastically reducing memory footprints.
- 3-Bit Performance Divergence: Vanilla rounding (RTN) fails catastrophically at 3 bits. However, advanced methods like AWQ, GPTQ, and SliM-LLM successfully maintain performance within 5% of the baseline.
- Ultra-Low Bit Limits: At ≤2 bits, standard methods fail completely. Specialized extreme-compression methods like BiLLM (at 1.1 bits) and DB-LLM (2 bits) manage to maintain reasonable text perplexity on LLaMA3-8B.
Detailed quantitative breakdowns for language perplexity can be seen in the evaluation tables below:
In terms of hardware deployment efficiency, AWQ's specialized quantization kernel achieved the highest inference speed on LLaMA3-8B, reaching 89.8 tokens per second.
Limitations and Future Work
The primary limitation highlighted by this study is the failure of current LoRA fine-tuning paradigms to improve quantized ultra-low bit models for highly pre-trained architectures like LLaMA3. The authors emphasize that the community needs to develop a new LoRA-FT paradigm specifically engineered for the LLaMA3 era—one that uses higher-quality datasets and alternative information-retention connections. Furthermore, protecting multimodal alignment from total collapse at 2 bits remains an open, challenging frontier for future MLLM quantization research.
Summary updated