TL;DR
Large Language Models (LLMs) are powerful but computationally expensive. This paper introduces AdaPruner, a novel framework that adaptively optimizes both the calibration data and importance estimation metrics used in structured pruning. This sample-aware approach significantly improves the efficiency and performance of pruned LLMs, achieving state-of-the-art results.
What Problem Does This Paper Address?
LLMs have revolutionized natural language processing, but their massive size and high computational demands hinder their practical deployment, especially in resource-constrained environments. Structured pruning offers a solution by removing redundant model parameters in a structured manner, making models more efficient without sacrificing too much performance. However, existing structured pruning methods often rely on randomly selected calibration data and fixed importance estimation metrics (like Taylor expansion). This can lead to suboptimal pruning decisions, resulting in degraded performance of the pruned models. The core challenge is that the quality of the calibration data directly impacts the accuracy of importance estimations, and a mismatch between the data and the chosen metric can lead to significant performance loss.
Methods
AdaPruner tackles this challenge by introducing a "sample-aware adaptive structured pruning" framework. The key innovation lies in its ability to adaptively search for the optimal combination of calibration data and importance estimation metrics.
Here's how it works:
- Structured Pruning Solution Space: AdaPruner defines a solution space that encompasses two interconnected subspaces:
- Calibration Data Subspace: This subspace is constructed using samples from the BookCorpus dataset, with an added filtering step to remove short, less informative texts, thereby accelerating the search.
- Importance Estimation Metrics Subspace: This subspace considers different granularities of importance estimation. It combines coarse-grained metrics (estimating the importance of entire weight structures) and fine-grained metrics (estimating the importance of individual weight elements). A weighted combination of these metrics is used, controlled by equilibrium coefficients ($\alpha_1, \alpha_2$).
- Bayesian Optimization: AdaPruner employs Bayesian Optimization (specifically, Tree-structured Parzen Estimator - BO-TPE) to efficiently search this solution space. BO-TPE is used to globally optimize the calibration data and importance estimation metrics simultaneously. It iteratively builds a probabilistic surrogate model to estimate the objective function (pruned model performance) and uses an acquisition function (Expected Improvement) to decide where to sample next, aiming to find the optimal configuration.
- Adaptive Pruning: Once the optimal calibration data and importance estimation metrics are identified, AdaPruner performs structured pruning on the LLM. To further enhance performance, the pruned models are fine-tuned using Low-Rank Adaptation (LoRA).
Datasets
- Calibration Data: BookCorpus dataset (filtered for longer texts).
- Evaluation Datasets:
- Language Modeling: WikiText2 and PTB.
- Commonsense Reasoning (Zero-shot): BoolQ, PIQA, HellaSwag, WinoGrande, ARC (easy and challenge), OpenbookQA.
- Fine-tuning Dataset: Stanford Alpaca dataset.
Key Insights and Ideas
- Interdependence of Calibration Data and Importance Metrics: The paper strongly emphasizes that calibration data and importance estimation metrics are not independent. Optimizing one without considering the other can lead to suboptimal results. AdaPruner's joint optimization approach is crucial for achieving superior performance.
- Randomness is Suboptimal: The study demonstrates that randomly selecting calibration data is suboptimal. Adaptive selection, as performed by AdaPruner, leads to significant performance improvements.
- Granularity Matters in Importance Estimation: Simply using coarse-grained or fine-grained importance metrics isn't enough. A balanced approach, combining both, is more effective.
- Bayesian Optimization for Efficiency: BO-TPE proves to be significantly more efficient than random search for finding optimal pruning configurations, converging much faster.
- Robustness Across Models and Pruning Ratios: AdaPruner shows consistent performance gains across different LLM architectures (LLaMA, Vicuna) and various pruning ratios, highlighting its generalization capabilities.
Key Results
- State-of-the-Art Performance: AdaPruner consistently outperforms existing structured pruning methods on LLaMA-7B and Vicuna-7B models.
- Performance Retention: At a 20% pruning ratio, AdaPruner-pruned models retain approximately 97% of the performance of the unpruned model.
- Significant Gains: On LLaMA-7B at 20% pruning, AdaPruner achieved an average performance of 61.44% across commonsense reasoning tasks, outperforming LLM-Pruner by 1.37%.
- Efficiency: AdaPruner's Bayesian optimization converges in under 150 iterations, significantly faster than random search.
- Ablation Studies: Removing the adaptive search for calibration data or importance metrics led to performance degradation, validating the core components of AdaPruner.
Limitations and Future Work
- Computational Cost of Optimization: While AdaPruner is more efficient than random search, the Bayesian optimization process itself still incurs computational overhead.
- Limited Pruning Solution Space: The current work focuses on specific types of calibration data and importance metrics. Future work could explore a broader range of pruning solution space designs.
- Model Types: The paper suggests exploring AdaPruner's potential for a wider variety of model types beyond the LLMs tested.
The authors propose that future work could investigate more sophisticated pruning solution space designs and extend AdaPruner to other model architectures.
Summary updated