TL;DR
This paper systematically compares pruning and quantization techniques for compressing Small Language Models (SLMs). It finds that quantization consistently outperforms pruning in preserving model fidelity and performance across various tasks and languages, especially under high compression. This suggests that for SLMs, quantization is the preferred method for efficient deployment.
What Problem Does This Paper Address?
Deploying language models on resource-constrained devices like mobile phones and wearables is a growing trend. While Large Language Models (LLMs) have been extensively studied for compression, Small Language Models (SLMs) are less explored. SLMs offer efficiency, faster inference, and lower memory footprints, making them ideal for edge devices. However, further compression is crucial to maximize their utility. This paper addresses the gap in understanding how state-of-the-art pruning and quantization methods, proven effective for LLMs, perform when applied to SLMs, and whether their effectiveness translates directly.
Methods
The authors conduct a comprehensive evaluation of two leading post-training pruning methods: SparseGPT and Wanda, and two leading quantization methods: GPTQ and AWQ. They evaluate these methods under two common compression settings: 50% sparsity (equivalent to 8-bit quantization) and 75% sparsity (equivalent to 4-bit quantization). The evaluation is performed across six SLMs ranging from 0.5B to 3.8B parameters, seven languages, and seven downstream tasks. The core evaluation metrics include:
- Compression Fidelity: Measured by Signal-to-Noise Ratio (SNR) and compression error of layer outputs.
- Language Modeling Capability: Assessed using perplexity across seven languages.
- Reasoning Capability: Evaluated through zero-shot performance on seven downstream natural language understanding tasks.
Datasets
The paper utilizes several publicly available datasets for its evaluation:
- Calibration Datasets: The initial shard of the C4 dataset and The Pile are used for calibration, following established methodologies for pruning and quantization methods.
- Multilingual Language Modeling: The mC4 dataset is used for evaluating perplexity across seven languages (English, Arabic, Hindi, Chinese, Thai, German, and Spanish).
- Downstream Tasks: The following datasets are used for zero-shot evaluation:
- ARC easy (ARC-e) and ARC challenge (ARC-c)
- BoolQ
- HellaSwag
- WinoGrande
- OpenBookQA
- RTE
Key Insights and Ideas
- Quantization Reigns Supreme for SLMs: The most striking finding is that quantization consistently outperforms pruning across all evaluation dimensions (fidelity, perplexity, and downstream tasks), especially under high compression. This challenges the notion that pruning might be equally effective or superior for smaller models.
- Pruning's Fragility in SLMs: Unlike in LLMs, where pruning methods like Wanda can be competitive, pruning techniques show significant degradation in SLMs, particularly at higher sparsity levels. This suggests that SLMs have less redundancy, making them more sensitive to weight removal.
- Wanda's Limitations for SLMs: The paper highlights that Wanda's assumptions, based on outlier features prevalent in LLMs, do not generalize well to SLMs. Wanda's heuristic approach, which doesn't explicitly minimize output differences, proves less effective for smaller models with potentially less pronounced outlier behavior.
- SparseGPT's Relative Strength in Pruning: While still underperforming quantization, SparseGPT generally shows better fidelity than Wanda in pruning SLMs, suggesting its second-order approximation might be more robust for smaller models.
- AWQ Leads Quantization: Among quantization methods, AWQ consistently shows slightly better results than GPTQ across models and tasks.
- Multilingual Performance Discrepancy: Pruning disproportionately affects resource-rich languages compared to typologically distinct languages (e.g., Arabic, Thai, Chinese), whereas quantization maintains more stable performance across language families.
Key Results
- Fidelity: Quantization methods (AWQ and GPTQ) achieve significantly higher SNR and lower compression error than pruning methods (SparseGPT and Wanda) across all models and compression ratios. For instance, at 75% sparsity, quantization maintains moderate fidelity (18-23 SNR), while pruning shows severe degradation (SNR around 1 for Qwen2.5).
- Language Modeling: Quantization consistently yields lower perplexity than pruning, especially at 75% sparsity. For example, on the R1 model, AWQ achieves a perplexity of 32.3 for English, while SparseGPT degrades to 292.4.
- Downstream Tasks: Quantization methods retain near-original accuracy with minimal drops (e.g., 1.81% at 50% compression), while pruning can lead to significant degradation (up to 16.61% at 50% compression and 48.07% at 75% sparsity).
- Cosine Similarity: Quantization methods preserve higher cosine similarity between compressed and original model hidden states, indicating better preservation of learned representations.
Limitations and Future Work
- Model Size Generalization: The study focuses on SLMs between 0.5B and 3.8B parameters. The direct applicability of these findings to other SLM sizes remains an open question, as compression dynamics might differ.
- Compression Methods: The study evaluates only four state-of-the-art post-training methods. Other compression techniques, particularly those requiring retraining, were not covered.
- Future Work: The authors suggest that further investigation into the impact of calibration dataset size and the exploration of other compression methods would be valuable. They also highlight the need to understand how compression effects might differ across various SLM architectures and training paradigms.
Summary updated