TL;DR
This paper provides a comprehensive evaluation of quantization techniques on large language models (LLMs) across various sizes and tasks. It reveals that while quantized models generally outperform smaller uncompressed models, they struggle with instruction following and hallucination detection, with FP8 emerging as the most robust quantization method.
What Problem Does This Paper Address?
Deploying large language models (LLMs) is computationally expensive, requiring significant memory and processing power. Quantization, a technique to reduce model size and computational overhead by using lower-precision numbers, offers a promising solution. However, most prior research has focused on older models and limited tasks, lacking a thorough evaluation of recent LLMs like Llama-3.3 across a broad spectrum of tasks and model sizes, from edge devices to giant models. This paper aims to fill this gap by systematically investigating the trade-offs between quantization methods, task difficulty, and model size.
Methods
The authors conducted a comprehensive evaluation using four post-training quantization (PTQ) methods: GPTQ, AWQ, SmoothQuant, and FP8. GPTQ and AWQ focus on weight-only quantization, while SmoothQuant and FP8 handle both weights and activations. The evaluation was performed on a multi-node cluster setup to ensure fast and reliable assessments across multiple benchmarks. They used the Im-eval and MT-Bench tools for evaluation, with MT-Bench employing LLMs as judges for free-form conversation quality.
Datasets
The study evaluated models on 13 datasets grouped into six categories:
- Commonsense Q&A: ARC, HellaSwag, Winogrande
- Knowledge and Language Understanding: MMLU, GPQA, MMLU-PRO, BBH, MuSR
- Instruction Following: IFEval
- Hallucination Detection: TruthfulQA
- Mathematics: GSM8K, MATH-Lvl-5
- Dialogue: MT-Bench
The evaluation also referenced results from the Huggingface OpenLLM Leaderboard-v1 and v2.
Key Insights and Ideas
- Quantization is a double-edged sword: While quantized models often outperform smaller FP16 baselines, they can falter on critical tasks like instruction following and hallucination detection. This suggests that quantization can amplify existing model weaknesses.
- FP8 reigns supreme: FP8 emerges as the most robust quantization method across all model sizes and tasks, particularly for very large models where other methods like SmoothQuant struggle. Hardware support for FP8 is a significant advantage.
- Size matters, but not always predictably: Smaller models (1B-3B) are highly susceptible to accuracy drops with 4-bit quantization, especially with GPTQ. However, 70B models maintain stable performance even at 4-bit. Interestingly, hard tasks don't always suffer the most significant accuracy degradation; the impact is more nuanced and depends on the model's inherent capabilities.
- Architecture and method interplay: Even at the same scale, different model architectures can exhibit varying quantization performance. AWQ consistently outperforms GPTQ in weight-only quantization.
- LLM-as-judge reveals hidden issues: While leaderboard metrics show general improvements, MT-Bench evaluations highlight significant performance declines in Coding and STEM tasks for quantized models, suggesting that traditional metrics might miss crucial qualitative degradations.
Key Results
- Quantized LLMs generally outperform smaller, uncompressed models across most benchmarks. For example, a 4-bit Llama-2-13B model performed better than an FP16 Llama-2-7B model.
- FP8 demonstrated superior stability and performance, especially for large models like Llama-3.1-405B.
- 4-bit quantization led to significant accuracy drops in smaller models (e.g., Llama-3.2-1B), with GPTQ showing particularly severe degradation.
- In Coding and STEM tasks, quantized models showed considerable performance reduction, while reasoning tasks sometimes saw improvements.
- MT-Bench evaluations indicated that quantized models suffered the largest score degradation in Coding and STEM categories.
Limitations and Future Work
The authors acknowledge that due to runtime limitations, they could not test all GPTQ configurations for the Llama-3.1-405B model. They also note that GPT4-based evaluations, while useful, can sometimes misjudge incorrect answers as correct, especially in complex reasoning or math tasks. For very large models and high-scoring categories, more sensitive metrics or superior judging models might be needed to discern subtle quality gaps. Future work could explore more advanced quantization techniques and develop more robust evaluation methodologies that better capture the nuances of quantized LLM performance.
Summary updated