An empirical study of LLaMA3 quantization: from LLMs to MLLMs

Wei Huang, Xingyu Zheng, Xudong Ma, Haotong Qin, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, Michele Magno

TL;DR

This paper presents a comprehensive empirical study benchmarking 10 different quantization techniques on Meta's LLaMA3 models (8B and 70B) and the LLaMA3-based multimodal model LLaVA-Next-8B. The authors reveal that while 4-bit quantization delivers near-lossless performance, ultra-low bit-widths (≤3 bits) cause severe degradation—particularly in multimodal tasks—and expose a surprising limitation where standard LoRA fine-tuning actually worsens quantization errors.

What Problem Does This Paper Address?

Large Language Models (LLMs) like LLaMA3 achieve state-of-the-art performance but demand massive computational and memory resources, making deployment on edge devices or resource-constrained environments highly challenging. Quantization—reducing the bit-width of model weights and activations (e.g., from 16-bit floating-point to 4-bit or 2-bit integers)—is a primary strategy to compress these models.

However, existing quantization benchmarks have primarily focused on older architectures like LLaMA1 and LLaMA2. Because LLaMA3 was pre-trained on a massive scale of over 15 trillion tokens, its internal knowledge structure and sensitivity to quantization noise differ significantly. This paper addresses the critical gap of understanding how modern, heavily pre-trained LLMs and Multi-Modal LLMs (MLLMs) behave under various low-bit compression methods.

Methods

The authors systematically evaluate two main technical tracks across a spectrum of 1 to 8 bits:

  1. Post-Training Quantization (PTQ): They evaluate 9 prominent weight-only and weight-activation PTQ methods: RTN (Round-to-Nearest), GPTQ, AWQ, SmoothQuant, PB-LLM, QuIP, DB-LLM, BiLLM, and SliM-LLM.
  2. LoRA Fine-Tuning (LoRA-FT): They evaluate 2 popular low-rank adaptation methods designed for quantized models: QLoRA and IR-QLoRA.

These methods are applied to LLaMA3-8B and LLaMA3-70B for text tasks, and to the LLM backbone of LLaVA-Next-8B for vision-language tasks.

Overview of the empirical study framework, mapping evaluated models to quantization methods and datasets.

Datasets

The quantization methods are validated across a diverse suite of benchmarks:

  • Language Perplexity (PPL): WikiText2, C4, and PTB.
  • Zero-Shot CommonSenseQA: PIQA, Winogrande, ARC-e, ARC-c, and HellaSwag.
  • Multi-Task Language Understanding: MMLU (specifically for evaluating LoRA fine-tuning).
  • Multimodal Vision-Language Tasks: AI2D, ChartQA, DocVQA, MMBench, and MME (Cognition and Perception).

Key Insights and Ideas

  • The LoRA Fine-Tuning Paradox: In a striking departure from LLaMA1 and LLaMA2 behavior, low-rank fine-tuning on the Alpaca dataset exacerbated performance degradation rather than fixing quantization errors. The authors hypothesize that because LLaMA3's base capabilities are so highly optimized by massive pre-training, fine-tuning on a small, generic dataset with limited parameters disrupts its sophisticated internal knowledge base.
  • Information Redundancy in Larger Scales: LLaMA3-70B demonstrated remarkable robustness against quantization compared to the 8B variant. As bit-width decreased, the 70B model's performance degraded much more slowly, confirming that larger models possess significant information redundancy that buffers against quantization noise.
  • The Multimodal Collapse: While 2-bit quantized models can still output coherent text sentences in pure language tasks, 2-bit quantization causes an absolute collapse in multimodal tasks (scores dropping to zero or generating repetitive gibberish). Visual-language alignment is exceptionally fragile and demands specialized mixed-precision handling (like SliM-LLM) to retain any utility at ultra-low bits.

Key Results

  • 4-Bit Quantization is Safe: Across almost all PTQ methods, 4-bit quantization incurs a negligible performance drop (around 2% compared to the uncompressed FP16 baseline) while drastically reducing memory footprints.
  • 3-Bit Performance Divergence: Vanilla rounding (RTN) fails catastrophically at 3 bits. However, advanced methods like AWQ, GPTQ, and SliM-LLM successfully maintain performance within 5% of the baseline.
  • Ultra-Low Bit Limits: At ≤2 bits, standard methods fail completely. Specialized extreme-compression methods like BiLLM (at 1.1 bits) and DB-LLM (2 bits) manage to maintain reasonable text perplexity on LLaMA3-8B.

Detailed quantitative breakdowns for language perplexity can be seen in the evaluation tables below:

Evaluation results of post-training quantization on the LLaMA3-8B model.
Evaluation results of post-training quantization on the LLaMA3-70B model.

In terms of hardware deployment efficiency, AWQ's specialized quantization kernel achieved the highest inference speed on LLaMA3-8B, reaching 89.8 tokens per second.

Limitations and Future Work

The primary limitation highlighted by this study is the failure of current LoRA fine-tuning paradigms to improve quantized ultra-low bit models for highly pre-trained architectures like LLaMA3. The authors emphasize that the community needs to develop a new LoRA-FT paradigm specifically engineered for the LLaMA3 era—one that uses higher-quality datasets and alternative information-retention connections. Furthermore, protecting multimodal alignment from total collapse at 2 bits remains an open, challenging frontier for future MLLM quantization research.

Summary updated

Abstract

The LLaMA family, a collection of foundation language models ranging from 7B to 65B parameters, has become one of the most powerful open-source large language models (LLMs) and the popular LLM backbone of multi-modal large language models (MLLMs), widely used in computer vision and natural language understanding tasks. In particular, LLaMA3 models have recently been released and have achieved impressive performance in various domains with super-large scale pre-training on over 15T tokens of data. Given the wide application of low-bit quantization for LLMs in resource-constrained scenarios, we explore LLaMA3's capabilities when quantized to low bit-width. This exploration can potentially provide new insights and challenges for the low-bit quantization of LLaMA3 and other future LLMs, especially in addressing performance degradation issues that suffer in LLM compression. Specifically, we comprehensively evaluate the 10 existing post-training quantization and LoRA fine-tuning (LoRA-FT) methods of LLaMA3 on 1-8 bits and various datasets to reveal the low-bit quantization performance of LLaMA3. To uncover the capabilities of low-bit quantized MLLM, we assessed the performance of the LLaMA3-based LLaVA-Next-8B model under 2-4 ultra-low bits with post-training quantization methods. Our experimental results indicate that LLaMA3 still suffers from non-negligible degradation in linguistic and visual contexts, particularly under ultra-low bit widths. This highlights the significant performance gap at low bit-width that needs to be addressed in future developments. We expect that this empirical study will prove valuable in advancing future models, driving LLMs and MLLMs to achieve higher accuracy at lower bit to enhance practicality. Our project is released on https://github.com/Macaronlin/LLaMA3-Quantization , and quantized models are released at https://huggingface.co/Efficient-ML .
BibTeX
@article{huang2024empirical,
  author = {Wei Huang and Xingyu Zheng and Xudong Ma and Haotong Qin and Chengtao Lv and Hong Chen and Jie Luo and Xiaojuan Qi and Xianglong Liu and Michele Magno},
  title = {{An empirical study of LLaMA3 quantization: from LLMs to MLLMs}},
  journal = {Vis. Intell.},
  year = {2024},
  doi = {10.1007/s44267-024-00070-x},
}
Figure