Abstract
The efficient compression of large language models (LLMs) has become increasingly popular. However, recovering the performance of compressed LLMs remains a major challenge. The current practice in LLM compression entails the implementation of structural pruning, complemented by a recovery phase that leverages the Low-Rank Adaptation (LoRA) algorithm. Structural pruning's uneven modification of model architecture, coupled with standard LoRA's fixed configuration allocation across layers in an online pipeline, leads to suboptimal performance in various downstream tasks for pruned models. To address this challenge, we introduce RankAdaptor, a hierarchical rank allocation method that enables efficient fine-tuning of pruned LLMs according to layerwise specific recovery requirements. We employ a performance model that conducts offline meta-learning and online incremental learning to explore optimal rank values for each layer. Comprehensive experiments on popular benchmarks show that RankAdaptor consistently outperforms state-of-the-art methods across a variety of pruning settings and LLM architectures, with improvements ranging from 0.7% to 5.5%.
Introduction
As Large Language Models (LLMs) continue to grow in size, the need for efficient compression techniques like structural pruning has become paramount for deployment on consumer-grade hardware. While structural pruning effectively reduces model size by removing entire neurons or blocks, it often results in a "destructive" modification of the architecture. This process creates structural irregularities where different layers retain varying degrees of semantic information and functional capacity.
The standard approach to recovering performance after pruning is Low-Rank Adaptation (LoRA). However, standard LoRA typically applies a uniform rank across all layers, failing to account for the specific recovery needs of an irregularly pruned model. We propose RankAdaptor, a framework that utilizes hierarchical rank allocation to provide customized recovery for each layer.
The Challenge: Structural Irregularity
Structural pruning does not affect every layer equally. Some layers may be more damaged or more critical for downstream task performance than others. Standard LoRA's "one-size-fits-all" approach ignores this reality. While adaptive methods like AdaLoRA attempt to adjust ranks during training (dynamic), they often struggle with the complex search space presented by pruned models. RankAdaptor addresses this by determining an optimal, static rank configuration before the final fine-tuning phase begins, ensuring that layers with higher recovery requirements receive more capacity.
The RankAdaptor Framework
RankAdaptor introduces a hierarchical rank allocation strategy powered by a predictive Performance Model. This model, a lightweight 5-layer Multi-Layer Perceptron (MLP), is designed to predict the downstream performance of a specific rank configuration.
The framework operates in three main phases:
- Offline Meta-learning: The performance model is pre-trained on diverse datasets to learn general relationships between rank distributions and model performance.
- Online Incremental Learning: For a specific task, the system samples a small number of rank configurations and performs mini-recovery trials. This data is used to fine-tune the performance model's predictions for that task.
- Optimal Selection: Once calibrated, the performance model efficiently scans thousands of candidate configurations to identify the optimal set of ranks ($R^*$) for the final recovery.
This approach allows RankAdaptor to navigate a mathematically vast search space (e.g., $6^{26}$ combinations for LLaMA-7B) without the prohibitive cost of full training cycles for every candidate.
Experimental Evaluation
We evaluated RankAdaptor using LLaMA-7B, LLaMA-13B, and Vicuna-7B across several zero-shot commonsense reasoning benchmarks, including BoolQ, PIQA, HellaSwag, WinoGrande, ARC, and OpenbookQA. We compared our method against standard LoRA and the adaptive AdaLoRA across multiple pruning methods (LLM-Pruner and Shortened) and rates (20% to 50%).
Key Results
RankAdaptor consistently outperformed both standard LoRA and AdaLoRA across all tested scenarios. As shown in the results table, the performance gap often widens at higher pruning rates, where the structural irregularity of the model is most pronounced.
For LLaMA-7B pruned at a 20% rate using LLM-Pruner, RankAdaptor achieved an average score of 60.38%, surpassing LoRA (59.09%) and AdaLoRA (58.53%). At a 50% pruning rate, the improvement remained significant, with RankAdaptor maintaining a lead of over 1.2% on average. Qualitatively, models recovered with RankAdaptor also demonstrated superior generative coherence in complex text tasks, providing clearer and more actionable outputs compared to standard LoRA.
Conclusion
RankAdaptor demonstrates that customized, hierarchical rank allocation is essential for the effective recovery of structurally pruned LLMs. By leveraging a lightweight performance proxy to navigate the configuration search space, RankAdaptor provides a scalable and highly effective alternative to uniform adaptation methods. Future work will explore the applicability of this hierarchical approach to other compression techniques, such as quantization.
Summary updated