Abstract
We present a simple meta quantization approach that quantizes different layers of a large language model (LLM) at different bit levels, and is independent of the underlying quantization technique. Specifically, we quantize the most important layers to higher bit precision and less important layers to lower bits. We propose two effective strategies to measure the importance of layers within LLMs: the first measures the importance of a layer based on how different its output embeddings are from the input embeddings (higher is better); the second estimates the importance of a layer using the number of layer weights that are much larger than average (smaller is better). We show that quantizing different layers at varying bits according to our importance scores results in minimal performance drop with a far more compressed model size. Finally, we present several practical key takeaways from our variable layer-wise quantization experiments: (a) LLM performance under variable quantization remains close to the original model until 25-50% of layers are moved in lower quantization using our proposed ordering but only until 5-10% if moved using no specific ordering; (b) Adding layer importance to inherently dynamic quantization techniques can further improve their performance, showing that our approach is complementary to other dynamic quantization methods; (c) Quantizing LLMs to lower bits performs substantially better than pruning unless extreme quantization (2-bit) is used; and (d) Layer-wise quantization to lower bits works better in the case of larger LLMs with more layers compared to smaller LLMs with fewer layers.
Introduction
As Large Language Models (LLMs) scale to hundreds of billions of parameters, their memory requirements become prohibitive for most users. For instance, a LLaMA3-8B model requires roughly 20GB of VRAM just to load at original precision—exceeding the capacity of many consumer GPUs. While quantization (reducing the precision of weights, e.g., from 16-bit to 4-bit) is a standard solution, most existing techniques apply a uniform bit-width across the entire model. This "blunt instrument" approach ignores a fundamental reality of neural networks: not all layers contribute equally to the model's output. This research addresses this inefficiency by proposing a method to selectively quantize layers based on their relative importance, enabling "fractional" bit-levels (e.g., 3.5 bits) that optimize the trade-off between model size and accuracy.
Methodology
The proposed meta-quantization approach is independent of the underlying quantization algorithm. The core of the method involves ranking layers by "importance" and keeping the most critical ones at higher precision (e.g., 4-bit) while moving others to lower precision (e.g., 2-bit).
To determine importance, two scoring metrics are introduced:
- Layer Input Modification (LIM): This measures how much a layer changes its input embeddings. If the output is significantly different from the input (low cosine similarity), the layer is deemed "important." This requires a small amount of unlabeled calibration data (e.g., 50 documents from the pg19 dataset).
- Z-Score Distribution (ZD): A data-free alternative that examines the distribution of weights within a layer. Layers with a higher proportion of "outlier" weights (weights far from the mean) are considered more important.
Experimental Evaluation
The researchers evaluated the method across five diverse LLM families: LLaMA-2 (7B, 13B), Mistral-7B, and Qwen (1.8B, 7B). Performance was measured on standard benchmarks including Winogrande, ARC-easy, PIQA, HellaSwag, and MMLU for reasoning, alongside GSM8K and Natural Questions for generation.
Key Results
The proposed layer-wise strategy allows models to maintain near-original performance down to an average of 3.0–3.25 bits. As shown in the accuracy curves for LLaMA-2-13B, LIM-based ordering (blue) retains performance significantly longer than random ordering (red) as the model bit size is reduced. Models using LIM ordering can often retain over 90% of their 4-bit performance even when several layers are dropped to 2-bit precision.
Key Insights and Discussion
- Quantization vs. Pruning: Quantizing layers to lower bits is almost always superior to removing them entirely (pruning), except at extreme compression levels below 2 bits.
- The "First and Last" Rule: Empirical evidence suggests that the very first and very last layers of an LLM are typically the most critical and should be prioritized for higher precision.
- Model Scale and Robustness: Larger models (like LLaMA-13B) handle non-uniform quantization better than smaller ones, likely due to a higher degree of layer redundancy.
- Mixing Strategies: Surprisingly, mixing just two bit-levels (e.g., 2 and 4) to reach a target average performs better than mixing three levels (e.g., 2, 4, and 8).
Limitations and Future Work
While effective, the method faces challenges in open-ended generation tasks (like GSM8K), which are more sensitive to quantization than classification tasks. Furthermore, while this method reduces VRAM requirements, current hardware kernels are often optimized for uniform bit-widths. Realizing the full inference speed benefits of fractional-bit models may require the development of custom kernels. Future work will explore more sophisticated layer-mixing strategies to further push the boundaries of LLM compression.
Summary updated