Abstract
Meta's LLaMA family has become one of the most powerful open-source Large Language Model (LLM) series. Notably, LLaMA3 models have recently been released and achieve impressive performance across various tasks with super-large scale pre-training on over 15T tokens of data. Given the wide application of low-bit quantization for LLMs in resource-limited scenarios, we explore LLaMA3's capabilities when quantized to low bit-width. This exploration holds the potential to unveil new insights and challenges for low-bit quantization of LLaMA3 and other forthcoming LLMs, especially in addressing performance degradation problems that suffer in LLM compression. Specifically, we evaluate the 10 existing post-training quantization and LoRA-finetuning methods of LLaMA3 on 1-8 bits and diverse datasets to comprehensively reveal LLaMA3's low-bit quantization performance. Our experiment results indicate that LLaMA3 still suffers non-negligent degradation in these scenarios, especially in ultra-low bit-width. This highlights the significant performance gap under low bit-width that needs to be bridged in future developments.
Introduction
The LLaMA3 family represents a massive leap in open-source AI, trained on a staggering 15 trillion tokens. However, as these models grow in "knowledge density," they also become more challenging to deploy on consumer hardware. This study provides a rigorous look at what happens when you compress LLaMA3 and its multi-modal cousin, LLaVA-Next, down to ultra-low bit-widths.
Experimental Framework
To understand the limits of LLaMA3 compression, the researchers conducted a massive sweep of the quantization landscape, evaluating both Large Language Models (LLMs) and Multi-modal Large Language Models (MLLMs).
Quantization Methods
The study benchmarks two primary approaches:
- Post-Training Quantization (PTQ): Ten established methods were tested, including GPTQ, AWQ, QuIP, and SmoothQuant, along with ultra-low bit specialists like BiLLM and DB-LLM.
- LoRA Fine-Tuning (LoRA-FT): QLoRA and IR-QLoRA were evaluated to determine if fine-tuning could recover accuracy lost during quantization.
Evaluation Benchmarks
Models were put through a gauntlet of benchmarks covering language modeling (WikiText2, C4, PTB), common sense reasoning (PIQA, ARC, HellaSwag, Winogrande), general knowledge (MMLU), and multi-modal vision tasks (AI2D, ChartQA, DocVQA, MME, MMBench).
Key Insights and Results
The 4-Bit Sweet Spot and 70B Resilience
For both 8B and 70B models, 4-bit quantization remains remarkably stable, with performance drops often limited to a negligible ~2%. Interestingly, the 70B model demonstrates significantly higher robustness to quantization than the 8B model, likely due to higher parameter redundancy that absorbs quantization noise more effectively.
The 2-Bit Performance Cliff
While 4-bit is stable, a significant "performance cliff" appears at 2-3 bits. For LLaMA3-8B, standard Round-To-Nearest (RTN) quantization at 2 bits causes perplexity to explode, rendering the model unusable.
As shown in the table above, specialized methods like QuIP manage to maintain some level of coherence (PPL 85.1 on WikiText2) where others fail completely, but the gap to full precision remains wide.
The LoRA-FT Paradox
A surprising finding is that LoRA fine-tuning on the Alpaca dataset actually worsened LLaMA3’s performance on MMLU compared to simple quantization. This suggests that LLaMA3’s pre-training is so comprehensive that fine-tuning on small, lower-quality datasets causes "knowledge forgetting."
Multi-modal Fragility
Vision-language tasks are even more sensitive. At 2 bits, LLaVA-Next-8B essentially collapses. On the MME Perception task, the 16-bit model scored 1588.3; while it stayed strong at 4-bit (1562.7), it dropped at 3-bit and failed entirely at 2-bit.
Conclusion
While 4-bit quantization is a viable path for deploying LLaMA3 on consumer hardware, ultra-low bit-widths (1-2 bits) remain a major challenge. The failure of standard LoRA-FT to improve results highlights a need for new fine-tuning paradigms and higher-quality calibration datasets. Future research must focus on bridging the performance gap at extreme compression levels to make these powerful models truly accessible.
Summary updated