TL;DR
This paper introduces FLAP, a novel, retraining-free method for structured pruning of Large Language Models (LLMs). FLAP effectively reduces model size and speeds up inference without requiring retraining, outperforming existing methods by intelligently identifying and removing redundant weights while compensating for performance loss.
What Problem Does This Paper Address?
Deploying massive LLMs like GPT-3 or LLaMA is computationally expensive, demanding significant resources for both training and inference. While model compression techniques like pruning aim to alleviate this, existing methods for LLMs often fall into two categories: unstructured pruning (which requires specialized hardware for speedups) or structured pruning that necessitates costly retraining. This paper tackles the challenge of achieving structured pruning – which removes entire rows or columns of weights, leading to more efficient hardware utilization – in a retraining-free manner, making LLM compression more accessible and practical.
Methods
FLAP is built upon three core components:
- Structured Fluctuation Metric: Instead of focusing on individual weights, FLAP measures the "fluctuation" of each input feature channel across different samples. The idea is that if a channel's output is stable and predictable (low fluctuation), pruning it is less likely to degrade performance. This metric helps determine if an output feature map can be recovered when a weight column is removed.
- Adaptive Structure Search: To avoid a one-size-fits-all approach, FLAP standardizes the fluctuation metric scores across different layers and modules. This allows for an adaptive search for the optimal compression structure, ensuring that pruning is tailored to the specific characteristics of each part of the LLM.
- Baseline Bias Compensation: This is the key to FLAP's retraining-free nature. After pruning, FLAP adds a bias term to the output feature maps. This bias is calculated based on the baseline values of the pruned channels, effectively compensating for the performance loss incurred by removing weights and recovering the model's accuracy without needing to retrain.
Datasets
The paper evaluates FLAP on the LLaMA model family (LLaMA-7B, 13B, 30B, 65B) and Vicuna-7B.
For language modeling performance, the WikiText2 validation set is used.
For zero-shot performance, the EleutherAI LM Harness is employed, covering seven common sense benchmarks: BoolQ, PIQA, HellaSwag, WinoGrande, ARC-easy, ARC-challenge, and OpenbookQA.
The paper also discusses the impact of calibration data selection, mentioning C4 and WikiText2 as examples.
Key Insights and Ideas
- Structured Pruning is Key for Hardware Efficiency: Unlike unstructured pruning, structured pruning directly translates to reduced memory footprint and faster inference on standard hardware, making it more practical for deployment.
- Retraining is a Bottleneck: The need for retraining in existing structured pruning methods is a major hurdle for LLMs due to their immense size and computational demands. FLAP's retraining-free approach is a significant advancement.
- Channel Stability Matters: The observation that certain hidden state channels exhibit stable behavior across samples is crucial. This stability allows for more aggressive pruning without severe performance degradation.
- Bias Compensation is a Powerful Alternative to Retraining: FLAP demonstrates that carefully designed bias terms can effectively recover performance lost during pruning, acting as a lightweight substitute for computationally expensive retraining or fine-tuning.
- Adaptive Structure is Crucial: Different layers and modules within an LLM have varying sensitivities to pruning. An adaptive approach that standardizes importance metrics and searches for optimal structures is essential for maximizing performance.
Key Results
- Outperforms State-of-the-Art: FLAP significantly outperforms existing methods like LLM-Pruner and Wanda-sp in structured pruning across various language benchmarks, even without any retraining.
- Language Modeling: On LLaMA-7B at a 50% pruning ratio, FLAP achieves a perplexity of 31.80, significantly better than LLM-Pruner (130.97 without LoRA, 39.02 with LoRA).
- Zero-Shot Performance: FLAP consistently achieves competitive or superior zero-shot performance compared to LLM-Pruner (with LoRA) across multiple benchmarks, especially at higher pruning ratios. For instance, on LLaMA-7B at 50% pruning, FLAP achieves 50.37 average accuracy, while LLM-Pruner (with LoRA) gets 48.98.
- Efficiency Gains: At a 20% pruning ratio, FLAP reduces parameters by 52% and accelerates inference speed by 66%. At 50% pruning, it achieves 25% parameter reduction and 31% speed increase.
- Robustness: The method shows good performance across different calibration datasets and is efficient, with the entire pruning process for LLaMA-7B completing in 3-5 minutes on a single GPU.
Limitations and Future Work
- Calibration Data Sensitivity: While the paper shows that overall accuracy differences are not significant with different calibration datasets, it acknowledges that specific datasets might suit different downstream tasks.
- Structured Attention Pruning: The paper notes that directly pruning rows/columns in self-attention modules is not straightforward and requires pruning at the "head" granularity. The current approach aggregates importance scores of neighboring weights for heads.
- Future Work: The authors hope their work contributes to a better understanding of structured pruning and performance recovery in LLMs, suggesting that bias compensation can largely replace retraining or Parameter-Efficient Fine-Tuning (PEFT). The framework's versatility suggests it can be extended to other Transformer-based LLMs.
Summary updated