Abstract
Large-scale pretraining followed by task-specific finetuning has achieved great success in various NLP tasks. Since finetuning all parameters of large pretrained models poses substantial computational and memory challenges, several efficient finetuning methods have been developed. Among them, low-rank adaptation (LoRA), which finetunes low-rank incremental update matrices on top of frozen pretrained weights, has proven particularly effective. Nonetheless, LoRA's uniform rank assignment across all layers, along with its reliance on an exhaustive search to find the best rank, leads to high computation costs and suboptimal finetuning performance. To address these limitations, we introduce AutoLoRA, a meta learning based framework for automatically identifying the optimal rank of each LoRA layer. AutoLoRA associates each rank-1 matrix in a low-rank update matrix with a selection variable, which determines whether the rank-1 matrix should be discarded. A meta learning based method is developed to learn these selection variables. The optimal rank is determined by thresholding the values of these variables. Our comprehensive experiments on natural language understanding, generation, and sequence labeling demonstrate the effectiveness of AutoLoRA.
Introduction
Fine-tuning Large Language Models (LLMs) like GPT-3 or RoBERTa is computationally intensive due to their massive parameter counts. Low-Rank Adaptation (LoRA) mitigates this by freezing the pretrained model weights and training only small, low-rank "update matrices." However, traditional LoRA faces two significant hurdles:
- Uniform Rank Assignment: It typically applies the same rank across all layers, ignoring the varying importance of different layers for specific tasks.
- Manual Hyperparameter Tuning: Finding the optimal rank requires exhaustive grid searches, which are slow and computationally expensive.
AutoLoRA addresses these issues by automating the rank selection process through a principled meta-learning optimization approach, making LoRA truly "plug-and-play."
The AutoLoRA Framework
AutoLoRA treats rank selection as a bi-level optimization problem. The core idea is to represent each LoRA update matrix as a sum of $k$ rank-1 matrices, each associated with a trainable selection variable ($\alpha$). These variables determine the importance of each rank-1 component.
The optimization process proceeds in two alternating steps:
- Weight Optimization: Update the model weights by minimizing loss on the training set.
- Rank Optimization: Update the selection variables ($\alpha$) by minimizing loss on a separate validation set.
To ensure stability, a softmax constraint is applied to the selection variables within each layer. Once the variables converge, they are thresholded to determine the final discrete rank for each layer, followed by a final retraining pass to optimize the model with the identified ranks.
Experimental Results
The effectiveness of AutoLoRA was evaluated across Natural Language Understanding (GLUE benchmark), Generation (E2E, WebNLG), and Sequence Labeling (BioNLP) tasks using RoBERTa and GPT-2 backbones.
As shown in the results, AutoLoRA achieved an average score of 85.5 on the GLUE benchmark, outperforming standard LoRA (84.9) and matching the performance of Full Fine-Tuning while using significantly fewer trainable parameters. Furthermore, AutoLoRA is highly efficient; it identifies optimal configurations in just 1.91x the baseline training time, compared to the 14.29x overhead typically required for an exhaustive grid search.
Key Insights
- Dynamic Layer Importance: Qualitative analysis reveals that different tasks require different ranks at various depths. For instance, in certain datasets, query projection layers often require higher ranks in the early and late blocks compared to the middle.
- Generalization via Meta-Learning: By tuning selection variables on a validation set rather than the training set, AutoLoRA avoids the overfitting issues seen in methods like AdaLoRA, leading to better generalization on unseen data.
- Efficient Search Space: The use of continuous selection variables allows the framework to explore the rank space using gradients, which is far more efficient than discrete search methods.
Conclusion and Future Work
AutoLoRA provides a robust and efficient framework for automating rank selection in low-rank adaptation. While it introduces some computational overhead compared to a single LoRA run, it drastically reduces the total time spent on hyperparameter tuning. Future work may explore automating the selection of layers for adaptation and scaling the framework to massive models with over 70 billion parameters.
Summary updated