RankAdaptor: Hierarchical Rank Allocation for Efficient Fine-Tuning Pruned LLMs via Performance Model

Changhai Zhou, Shijie Han, S. Zhang, Shichao Weng, Zekai Liu, Cheng Jin, Li, Hongguang

Abstract

The efficient compression of large language models (LLMs) has become increasingly popular. However, recovering the performance of compressed LLMs remains a major challenge. The current practice in LLM compression entails the implementation of structural pruning, complemented by a recovery phase that leverages the Low-Rank Adaptation (LoRA) algorithm. Structural pruning's uneven modification of model architecture, coupled with standard LoRA's fixed configuration allocation across layers in an online pipeline, leads to suboptimal performance in various downstream tasks for pruned models. To address this challenge, we introduce RankAdaptor, a hierarchical rank allocation method that enables efficient fine-tuning of pruned LLMs according to layerwise specific recovery requirements. We employ a performance model that conducts offline meta-learning and online incremental learning to explore optimal rank values for each layer. Comprehensive experiments on popular benchmarks show that RankAdaptor consistently outperforms state-of-the-art methods across a variety of pruning settings and LLM architectures, with improvements ranging from 0.7% to 5.5%.

Introduction

To facilitate the deployment of Large Language Models (LLMs) on consumer-grade hardware, structural pruning is frequently employed to remove redundant neurons and connections. While effective for compression, pruning is inherently non-uniform; it modifies layers to varying degrees, creating structural irregularities that degrade model performance.

The standard recovery protocol involves fine-tuning the pruned model using Low-Rank Adaptation (LoRA). However, conventional LoRA typically applies a fixed rank across all layers. This "one-size-fits-all" strategy is suboptimal because heavily pruned layers often require greater adaptation capacity (higher ranks) than those less affected. Determining the optimal rank for each layer manually is computationally infeasible, given the astronomical search space (e.g., $6^{32}$ combinations for a 32-layer model).

Comparison of LoRA vs RankAdaptor in the structural pruning workflow.

Methodology: The RankAdaptor Framework

RankAdaptor replaces fixed rank allocation with a hierarchical, layer-wise approach guided by a predictive Performance Model. This model is a lightweight 5-layer Multi-Layer Perceptron (MLP) designed to estimate the performance of a pruned LLM given a specific rank configuration vector.

The framework operates in two distinct learning phases:

  1. Offline Meta-learning: The Performance Model is pre-trained on diverse datasets to capture general heuristics regarding how different rank distributions influence overall performance.
  2. Online Incremental Learning: Upon selecting a specific downstream task (e.g., BoolQ or HellaSwag), the model is fine-tuned on a small subset of actual training data. This specialization allows the model to accurately predict task-specific performance.

Once trained, the Performance Model rapidly evaluates thousands of candidate configurations to identify the optimal hierarchical rank distribution. The pruned LLM is then fine-tuned a single time using this optimized setup.

Experimental Setup and Results

The researchers evaluated RankAdaptor across seven commonsense reasoning benchmarks, including BoolQ, PIQA, and ARC, using architectures such as LLaMA-7B/13B and Vicuna-7B. Two pruning strategies, LLM-Pruner and Shortened LLaMA, were tested at ratios ranging from 20% to 50%.

Performance Gains

RankAdaptor consistently outperformed both standard LoRA and AdaLoRA. Notably, AdaLoRA—which dynamically adjusts ranks—was found to perform worse than standard LoRA on pruned models, likely due to the complexity of the pruned architecture confusing its importance estimation.

Detailed performance comparison of RankAdaptor against LoRA and AdaLoRA across various pruning rates and methods.

Key findings include:

  • Accuracy Improvements: RankAdaptor achieved gains of 0.7% to 5.5% over baseline methods.
  • Robustness: The method maintained superior performance even at high (50%) pruning rates, where model integrity typically degrades significantly.
  • Qualitative Quality: Beyond metrics, RankAdaptor-recovered models produced more coherent and logical text in generative tasks compared to those fine-tuned with fixed-rank LoRA.

Conclusion

RankAdaptor demonstrates that hierarchical rank allocation is critical for recovering the performance of structurally pruned LLMs. By leveraging a predictive performance model, the framework bypasses the prohibitive cost of manual configuration search. While the current work focuses on pruning, future research will explore the synergy between RankAdaptor and other compression techniques like quantization, as well as methods to further reduce the initial data requirements for the performance model's incremental learning phase.

Summary updated

Abstract

The efficient compression of large language models (LLMs) has become increasingly popular. However, recovering the performance of compressed LLMs remains a major challenge. The current practice in LLM compression entails the implementation of structural pruning, complemented by a recovery phase that leverages the Low-Rank Adaptation (LoRA) algorithm. Structural pruning's uneven modification of model architecture, coupled with standard LoRA's fixed configuration allocation across layers in an online pipeline, leads to suboptimal performance in various downstream tasks for pruned models. To address this challenge, we introduce RankAdaptor, a hierarchical rank allocation method that enables efficient fine-tuning of pruned LLMs according to layerwise specific recovery requirements. We employ a performance model that conducts offline meta-learning and online incremental learning to explore optimal rank values for each layer. Comprehensive experiments on popular benchmarks show that RankAdaptor consistently outperforms state-of-the-art methods across a variety of pruning settings and LLM architectures, with improvements ranging from 0.7\% to 5.5\%.
BibTeX
@inproceedings{zhou2025rankadaptor,
  author = {Changhai Zhou and Shijie Han and S. Zhang and Shichao Weng and Zekai Liu and Cheng Jin and Li, Hongguang},
  title = {{RankAdaptor: Hierarchical Rank Allocation for Efficient Fine-Tuning Pruned LLMs via Performance Model}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.findings-naacl.321},
}
Figure