Abstract
The LLaMA family, a collection of foundation language models ranging from 7B to 65B parameters, has become one of the most powerful open-source large language models (LLMs) and the popular LLM backbone of multi-modal large language models (MLLMs). In particular, LLaMA3 models have recently been released and have achieved impressive performance in various domains with super-large scale pre-training on over 15T tokens of data. Given the wide application of low-bit quantization for LLMs in resource-constrained scenarios, we explore LLaMA3's capabilities when quantized to low bit-width. Specifically, we comprehensively evaluate the 10 existing post-training quantization and LoRA fine-tuning (LoRA-FT) methods of LLaMA3 on 1-8 bits and various datasets. To uncover the capabilities of low-bit quantized MLLM, we assessed the performance of the LLaMA3-based LLaVA-Next-8B model under 2-4 ultra-low bits. Our experimental results indicate that LLaMA3 still suffers from non-negligible degradation in linguistic and visual contexts, particularly under ultra-low bit widths. This highlights the significant performance gap at low bit-width that needs to be addressed in future developments.
Introduction
Large Language Models (LLMs) like LLaMA3 are incredibly capable but notoriously resource-hungry, making them difficult to deploy on consumer hardware or edge devices. Quantization—the process of reducing the numerical precision of model weights (e.g., from 16-bit floating point to 4-bit integers)—is the standard solution to reduce memory footprint and speed up inference. However, LLaMA3 was trained on a massive 15 trillion tokens, resulting in a model with much higher "information density" than its predecessors. This study provides a comprehensive "stress test" to determine if this increased density makes LLaMA3 more or less resilient to compression, particularly when acting as the "brain" for Multi-modal Large Language Models (MLLMs).
Methodology
The study evaluates LLaMA3-8B, LLaMA3-70B, and the multi-modal LLaVA-Next-8B across two primary tracks: Post-Training Quantization (PTQ) and LoRA Fine-Tuning (LoRA-FT).
Ten PTQ algorithms were tested, including industry standards like GPTQ and AWQ, alongside specialized ultra-low bit methods like BiLLM and QuIP. For LoRA-FT, the researchers examined QLoRA and IR-QLoRA. Evaluation metrics included Perplexity (PPL) on WikiText2, C4, and PTB datasets, zero-shot benchmarks for common sense reasoning, and specialized multi-modal benchmarks (AI2D, ChartQA, etc.) for visual understanding.
Quantization Performance and Results
Across almost all methods, 4-bit quantization emerged as the "sweet spot," resulting in a negligible performance loss of approximately 2%. This makes 4-bit the clear choice for practical deployment. However, a significant "performance cliff" is observed at ultra-low bit-widths (2-3 bits).
Key Insights
- Model Size and Redundancy: LLaMA3-70B is significantly more robust to quantization than the 8B version, suggesting that larger models possess more redundant parameters that can be compressed without losing core logic.
- The LoRA Paradox: Surprisingly, fine-tuning LLaMA3-8B on the Alpaca dataset (LoRA-FT) worsened performance at low bit-widths compared to the base quantized model. The researchers hypothesize that LLaMA3's high optimization on 15T tokens makes it sensitive to "distractions" from smaller, lower-quality datasets during fine-tuning.
- Multi-modal Fragility: When used in an MLLM like LLaVA-Next, LLaMA3 is much more sensitive to quantization. At 2 bits, the model often collapses, failing to identify basic objects or outputting repetitive text.
- Efficiency: AWQ proved highly effective for practical use, achieving an inference speed of 89.8 tokens/second on LLaMA3-8B while maintaining high accuracy.
Conclusion
While LLaMA3 remains remarkably robust at 4 bits—with a 4-bit LLaMA3-8B even outperforming a full-precision LLaMA2-7B—ultra-low bit quantization (2-bit and below) remains an unsolved challenge. The observed performance gap suggests a need for new quantization paradigms, potentially those that are aware of the multi-modal training process, to bridge the gap between efficiency and intelligence in future models.
Summary updated