Abstract
We present a simple meta quantization approach that quantizes different layers of a large language model (LLM) at different bit levels, and is independent of the underlying quantization technique. Specifically, we quantize the most important layers to higher bit precision and less important layers to lower bits. We propose two effective strategies to measure the importance of layers within LLMs: the first measures the importance of a layer based on how different its output embeddings are from the input embeddings (higher is better); the second estimates the importance of a layer using the number of layer weights that are much larger than average (smaller is better). We show that quantizing different layers at varying bits as per our importance scores results in minimal performance drop with a far more compressed model. Finally, we present several practical key takeaways from our variable layer-wise quantization experiments: (a) LLM performance under variable quantization remains close to the original model until 25–50% of layers are moved in lower quantization using our proposed ordering but only until 5–10% if moved using no specific ordering; (b) Adding layer importance to inherently dynamic quantization techniques can further improve their performance, showing that our approach is complementary to other dynamic quantization methods; (c) Quantizing LLMs to lower bits performs substantially better than pruning unless extreme quantization (2-bit) is used; and (d) Layer-wise quantization to lower bits works better in the case of larger LLMs with more layers compared to smaller LLMs with fewer layers.
Introduction
Large Language Models (LLMs) are notoriously memory-hungry. Even a "small" model like LLaMa3-8B requires approximately 20GB of VRAM at original precision, creating a significant barrier for researchers and edge device deployment. While quantization—reducing the bit-width of weights—is a standard solution, most existing methods apply a uniform bit-width (e.g., 4-bit) across the entire network.
This paper argues that uniform quantization is inefficient. Deep learning research suggests that different layers contribute disproportionately to a model's final output. By treating every layer identically, we either waste memory on redundant layers or sacrifice accuracy by over-compressing critical ones. The authors propose a method to identify and preserve the most "important" layers at higher precision while aggressively compressing the rest.
Methodology: Variable Layerwise Quantization
The researchers introduce Variable Layerwise Quantization, a "meta" method compatible with existing techniques like GPT-Q or Quanto. The strategy involves two primary steps: ranking layers by importance and assigning bit-widths based on that rank.
Two novel strategies are proposed to measure layer importance:
- Layer Input Modification (LIM): This measures the transformation a layer applies to its input. If output embeddings differ significantly from input embeddings (measured via negative cosine similarity), the layer is deemed critical. This requires a small, unlabeled calibration dataset (e.g., 50 samples from PG19).
- Z-score Distribution (ZD): A data-free method that analyzes weight distributions. It calculates the ratio of "outlier" weights (z-score > 1). Layers with more outliers are considered harder to quantize and thus more important to keep at higher precision.
Once ranked, the top $X$ layers are kept at higher precision (e.g., 4-bit), while the remaining layers are dropped to lower precision (e.g., 2-bit).
Experimental Results
The approach was evaluated across diverse tasks including Winogrande, ARC-easy, PIQA, HellaSwag, and MMLU using models such as LLaMa-2, Mistral, and Qwen.
Key Findings
- Performance Retention: Models maintained near-original performance until roughly 25–50% of layers were moved to lower quantization using the proposed importance ordering. In contrast, random ordering caused performance to collapse after moving only 5–10% of layers.
- Extreme Compression: The authors successfully compressed models to an average of 3.0–3.25 bits with negligible loss. For LLaMa2-13B, they reached 2.85 bits while retaining 90% of the 4-bit model's accuracy.
- Memory Efficiency: For Qwen-7B, moving 90% of layers to 2-bit precision (guided by LIM) reduced the VRAM footprint by 36.5% compared to a uniform 4-bit baseline.
Key Insights and Discussion
- The "U-Shaped" Importance Curve: Analysis shows that the first and last layers are typically the most critical, while "middle" layers are often more redundant and tolerant of heavy compression.
- Quantization vs. Pruning: For most memory budgets, it is superior to keep a layer at a very low bit-width (2-bit) than to prune it entirely, unless extreme compression (< 3 bits) is required.
- Model Scale and Resilience: Larger models (e.g., LLaMa-2-13B) exhibit better performance retention than smaller models (e.g., Qwen-1.8B), suggesting that increased scale provides internal redundancy that can be exploited for compression.
- Simplicity Wins: Surprisingly, quantizing layers into just two buckets (e.g., 4-bit and 2-bit) performed better than spreading them across three levels (8-bit, 4-bit, and 2-bit) for the same total memory footprint.
Conclusion
Variable Layerwise Quantization offers a simple yet powerful framework for LLM compression. By leveraging unsupervised or data-free importance metrics, researchers can achieve significant memory savings without the catastrophic performance loss associated with uniform low-bit quantization or pruning. Future work may explore supervised importance metrics tailored to specific downstream tasks.
Summary updated