Abstract
Large-scale pretraining followed by task-specific finetuning has achieved great success in various NLP tasks. Since finetuning all parameters of large pretrained models poses substantial computational and memory challenges, several efficient finetuning methods have been developed. Among them, low-rank adaptation (LoRA), which finetunes low-rank incremental update matrices on top of frozen pretrained weights, has proven particularly effective. Nonetheless, LoRA’s uniform rank assignment across all layers, along with its reliance on an exhaustive search to find the best rank, leads to high computation costs and suboptimal finetuning performance. To address these limitations, we introduce AutoLoRA, a meta learning based framework for automatically identifying the optimal rank of each LoRA layer. AutoLoRA associates each rank-1 matrix in a low-rank update matrix with a selection variable, which determines whether the rank-1 matrix should be discarded. A meta learning based method is developed to learn these selection variables. The optimal rank is determined by thresholding the values of these variables. Our comprehensive experiments on natural language understanding, generation, and sequence labeling demonstrate the effectiveness of AutoLoRA.
Introduction
Fine-tuning Large Language Models (LLMs) typically involves updating billions of parameters, a process that is computationally prohibitive for many researchers. Low-Rank Adaptation (LoRA) revolutionized this space by freezing the original model weights and only training small, low-rank update matrices. However, LoRA traditionally employs a "one-size-fits-all" approach, assigning the same rank to every layer. This requires manual, exhaustive grid searches to find the best global rank, which is both time-consuming and likely suboptimal, as different layers contribute differently to specific downstream tasks.
AutoLoRA addresses these challenges by using meta-learning to automatically determine layer-specific ranks, eliminating the need for manual tuning while improving adaptation performance.
Methodology: The AutoLoRA Framework
AutoLoRA utilizes a bi-level optimization strategy rooted in meta-learning to identify the importance of individual rank components. The process begins by decomposing the LoRA update matrix into a summation of $k$ rank-1 matrices. Each of these components is paired with a trainable "selection variable" ($\alpha$), which acts as a gate for that specific component's contribution.
The framework operates through an iterative loop:
- Weight Optimization: The model's low-rank update matrices are trained on the primary training set.
- Rank Optimization: The selection variables ($\alpha$) are updated by minimizing loss on a separate validation set. This ensures that the importance scores reflect the model's ability to generalize rather than just its ability to memorize training data.
- Rank Determination: Once converged, selection variables below a specific threshold are pruned. The model is then retrained using the resulting optimal, layer-specific rank configuration.
Experimental Results
We evaluated AutoLoRA across a diverse set of benchmarks, including Natural Language Understanding (GLUE), Natural Language Generation (E2E, WebNLG), and Sequence Labeling (BioNLP), using models such as RoBERTa and GPT-2.
Performance on NLU Tasks
On the GLUE benchmark, AutoLoRA demonstrated superior performance compared to standard LoRA and other adaptive methods like AdaLoRA. Notably, AutoLoRA achieved an average score of 85.5, matching the performance of Full Fine-Tuning while only updating approximately 0.2% of the total parameters (0.3M vs. 125M).
Efficiency and Robustness
Beyond raw performance, AutoLoRA offers significant efficiency gains. Finding the optimal configuration with AutoLoRA was approximately 7.5 times faster than performing a standard 16-configuration grid search for LoRA. Qualitative analysis of the learned ranks revealed that AutoLoRA tends to assign higher ranks to the final layers of the model, confirming the hypothesis that different layers have varying levels of importance for downstream tasks.
Conclusion
AutoLoRA provides a robust, automated solution to the rank-selection problem in low-rank adaptation. By shifting from discrete, manual searches to a continuous, gradient-based meta-learning framework, it enables more efficient and effective fine-tuning of large-scale models. Future work will explore extending this automation to layer selection itself and testing the framework's efficacy across a broader range of multilingual tasks.
Summary updated