An empirical study of LLaMA3 quantization: from LLMs to MLLMs

Wei Huang, Xingyu Zheng, Xudong Ma, Haotong Qin, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, Michele Magno

Abstract

The LLaMA family, a collection of foundation language models ranging from 7B to 65B parameters, has become one of the most powerful open-source large language models (LLMs) and the popular LLM backbone of multi-modal large language models (MLLMs). In particular, LLaMA3 models have recently been released and have achieved impressive performance in various domains with super-large scale pre-training on over 15T tokens of data. Given the wide application of low-bit quantization for LLMs in resource-constrained scenarios, we explore LLaMA3's capabilities when quantized to low bit-width. Specifically, we comprehensively evaluate the 10 existing post-training quantization and LoRA fine-tuning (LoRA-FT) methods of LLaMA3 on 1-8 bits and various datasets. To uncover the capabilities of low-bit quantized MLLM, we assessed the performance of the LLaMA3-based LLaVA-Next-8B model under 2-4 ultra-low bits. Our experimental results indicate that LLaMA3 still suffers from non-negligible degradation in linguistic and visual contexts, particularly under ultra-low bit widths. This highlights the significant performance gap at low bit-width that needs to be addressed in future developments.

Introduction

Large Language Models (LLMs) like LLaMA3 are incredibly capable but notoriously resource-hungry, making them difficult to deploy on consumer hardware or edge devices. Quantization—the process of reducing the numerical precision of model weights (e.g., from 16-bit floating point to 4-bit integers)—is the standard solution to reduce memory footprint and speed up inference. However, LLaMA3 was trained on a massive 15 trillion tokens, resulting in a model with much higher "information density" than its predecessors. This study provides a comprehensive "stress test" to determine if this increased density makes LLaMA3 more or less resilient to compression, particularly when acting as the "brain" for Multi-modal Large Language Models (MLLMs).

Methodology

The study evaluates LLaMA3-8B, LLaMA3-70B, and the multi-modal LLaVA-Next-8B across two primary tracks: Post-Training Quantization (PTQ) and LoRA Fine-Tuning (LoRA-FT).

Overview of the LLaMA3 quantization evaluation framework, including models, methods, and datasets.

Ten PTQ algorithms were tested, including industry standards like GPTQ and AWQ, alongside specialized ultra-low bit methods like BiLLM and QuIP. For LoRA-FT, the researchers examined QLoRA and IR-QLoRA. Evaluation metrics included Perplexity (PPL) on WikiText2, C4, and PTB datasets, zero-shot benchmarks for common sense reasoning, and specialized multi-modal benchmarks (AI2D, ChartQA, etc.) for visual understanding.

Quantization Performance and Results

Across almost all methods, 4-bit quantization emerged as the "sweet spot," resulting in a negligible performance loss of approximately 2%. This makes 4-bit the clear choice for practical deployment. However, a significant "performance cliff" is observed at ultra-low bit-widths (2-3 bits).

Perplexity results for LLaMA3-8B across different quantization methods and bit-widths, illustrating the performance cliff below 4 bits.

Key Insights

  • Model Size and Redundancy: LLaMA3-70B is significantly more robust to quantization than the 8B version, suggesting that larger models possess more redundant parameters that can be compressed without losing core logic.
  • The LoRA Paradox: Surprisingly, fine-tuning LLaMA3-8B on the Alpaca dataset (LoRA-FT) worsened performance at low bit-widths compared to the base quantized model. The researchers hypothesize that LLaMA3's high optimization on 15T tokens makes it sensitive to "distractions" from smaller, lower-quality datasets during fine-tuning.
  • Multi-modal Fragility: When used in an MLLM like LLaVA-Next, LLaMA3 is much more sensitive to quantization. At 2 bits, the model often collapses, failing to identify basic objects or outputting repetitive text.
  • Efficiency: AWQ proved highly effective for practical use, achieving an inference speed of 89.8 tokens/second on LLaMA3-8B while maintaining high accuracy.

Conclusion

While LLaMA3 remains remarkably robust at 4 bits—with a 4-bit LLaMA3-8B even outperforming a full-precision LLaMA2-7B—ultra-low bit quantization (2-bit and below) remains an unsolved challenge. The observed performance gap suggests a need for new quantization paradigms, potentially those that are aware of the multi-modal training process, to bridge the gap between efficiency and intelligence in future models.

Summary updated

Abstract

The LLaMA family, a collection of foundation language models ranging from 7B to 65B parameters, has become one of the most powerful open-source large language models (LLMs) and the popular LLM backbone of multi-modal large language models (MLLMs), widely used in computer vision and natural language understanding tasks. In particular, LLaMA3 models have recently been released and have achieved impressive performance in various domains with super-large scale pre-training on over 15T tokens of data. Given the wide application of low-bit quantization for LLMs in resource-constrained scenarios, we explore LLaMA3’s capabilities when quantized to low bit-width. This exploration can potentially provide new insights and challenges for the low-bit quantization of LLaMA3 and other future LLMs, especially in addressing performance degradation issues that suffer in LLM compression. Specifically, we comprehensively evaluate the 10 existing post-training quantization and LoRA fine-tuning (LoRA-FT) methods of LLaMA3 on 1-8 bits and various datasets to reveal the low-bit quantization performance of LLaMA3. To uncover the capabilities of low-bit quantized MLLM, we assessed the performance of the LLaMA3-based LLaVA-Next-8B model under 2-4 ultra-low bits with post-training quantization methods. Our experimental results indicate that LLaMA3 still suffers from non-negligible degradation in linguistic and visual contexts, particularly under ultra-low bit widths. This highlights the significant performance gap at low bit-width that needs to be addressed in future developments. We expect that this empirical study will prove valuable in advancing future models, driving LLMs and MLLMs to achieve higher accuracy at lower bit to enhance practicality.
BibTeX
@article{huang2024empirical,
  author = {Wei Huang and Xingyu Zheng and Xudong Ma and Haotong Qin and Chengtao Lv and Hong Chen and Jie Luo and Xiaojuan Qi and Xianglong Liu and Michele Magno},
  title = {{An empirical study of LLaMA3 quantization: from LLMs to MLLMs}},
  journal = {Vis. Intell.},
  year = {2024},
  doi = {10.1007/s44267-024-00070-x},
}
Figure