OP-LoRA: The Blessing of Dimensionality

Piotr Teterwak, Kate Saenko, Bryan A. Plummer, Ser-Nam Lim

Abstract

Low-rank adapters enable fine-tuning of large models with only a small number of parameters, thus reducing storage costs and minimizing the risk of catastrophic forgetting. However, they often pose optimization challenges, with poor convergence. To overcome these challenges, we introduce an over-parameterized approach that accelerates training without increasing inference costs. This method reparameterizes low-rank adaptation by employing a separate MLP and learned embedding for each layer. The learned embedding is input to the MLP, which generates the adapter parameters. Such overparamaterization has been shown to implicitly function as an adaptive learning rate and momentum, accelerating optimization. At inference time, the MLP can be discarded, leaving behind a standard low-rank adapter. To study the effect of MLP overparameterization on a small yet difficult proxy task, we implement it for matrix factorization, and find it achieves faster convergence and lower final loss. Extending this approach to larger-scale tasks, we observe consistent performance gains across domains. We achieve improvements in vision-language tasks and especially notable increases in image generation, with CMMD scores improving by up to 15 points.

Introduction

Fine-tuning massive foundation models has become a cornerstone of modern AI, yet it remains a resource-intensive challenge. Low-Rank Adaptation (LoRA) has emerged as the industry standard by updating only a small fraction of a model's weights, significantly reducing storage and deployment overhead. However, LoRA's efficiency comes with a trade-off: the optimization landscape of low-rank matrices is often poorly conditioned, leading to slow convergence and high sensitivity to hyperparameters. This paper explores a counter-intuitive solution: improving optimization by adding parameters during training that are discarded before inference.

The OP-LoRA Framework

We introduce OP-LoRA (Over-Parameterized LoRA), a method that reparameterizes low-rank adaptation to accelerate training without increasing inference costs. In a standard LoRA setup, two small matrices, $A$ and $B$, are learned directly. In contrast, OP-LoRA employs a compact Multi-Layer Perceptron (MLP) and a learned embedding for each layer.

Overview of OP-LoRA architecture and performance comparison.

During training, the learned embedding is processed by the MLP to generate the weights for the $A$ and $B$ adapter matrices. This overparameterization acts as an implicit optimizer, providing effects similar to an adaptive learning rate and momentum. This "blessing of dimensionality" allows the model to navigate the optimization landscape more effectively, escaping local minima that might stall standard LoRA. Crucially, once training is complete, the MLP and embeddings are used one final time to produce the static $A$ and $B$ weights. The MLP is then discarded, resulting in a standard LoRA adapter with zero additional computational or storage overhead at inference time.

Experimental Evaluation

The efficacy of OP-LoRA was evaluated across a diverse set of tasks, ranging from synthetic proxy problems to large-scale vision and language benchmarks.

  • Matrix Factorization: On a $100 \times 100$ matrix factorization task, OP-LoRA consistently reached the optimal SVD (Singular Value Decomposition) solution, whereas standard LoRA-style approaches frequently stalled at higher error rates.
  • Image Generation: The most significant gains were observed in image generation. Fine-tuning Stable Diffusion XL on the Naruto BLIP dataset, OP-LoRA improved the CMMD score (where lower is better) from 23.7 to 9.6. Qualitative results showed much higher faithfulness to complex prompts, such as correctly rendering specific clothing items like hoodies.
  • Vision-Language and Reasoning: Across benchmarks like VQAv2, GQA, and NLVRv2 using VL-BART, as well as commonsense reasoning with LLaMA-7B, OP-LoRA (and its variant OP-DoRA) consistently outperformed standard baselines by 1-4%.
  • Image Classification: Tested on LLaVA-1.5-7B with the Stanford Cars dataset, the method showed robust performance gains.

Discussion and Future Work

The success of OP-LoRA highlights the "blessing of dimensionality" in deep learning optimization. By providing more flexible paths for gradient flow, the MLP-based overparameterization smooths the optimization landscape. This allows the model to overcome the low-rank bias inherent in standard LoRA, capturing "minor" details of a task by effectively jumping to a higher rank during training before settling into the final low-rank solution. Furthermore, this approach makes the training process significantly more robust to hyperparameter choices, such as the learning rate, making it a more "plug-and-play" solution for engineers.

Despite its advantages, OP-LoRA introduces a slight increase in memory usage during the training phase due to the MLP parameters. Additionally, the architecture of the MLP—specifically its width—requires careful tuning, as the researchers observed an "inverted U-shape" relationship where both too small and too large MLPs lead to suboptimal performance. Future research will focus on scaling these techniques to ultra-large models (e.g., 70B+ parameters) and automating the selection of MLP architectures.

Summary updated

Abstract

Low-rank adapters enable fine-tuning of large models with only a small number of parameters, thus reducing storage costs and minimizing the risk of catastrophic forgetting. However, they often pose optimization challenges, with poor convergence. To overcome these challenges, we introduce an over-parameterized approach that accelerates training without increasing inference costs. This method reparameterizes low-rank adaptation by employing a separate MLP and learned embedding for each layer. The learned embedding is input to the MLP, which generates the adapter parameters. Such overparamaterization has been shown to implicitly function as an adaptive learning rate and momentum, accelerating optimization. At inference time, the MLP can be discarded, leaving behind a standard low-rank adapter. To study the effect of MLP overparameterization on a small yet difficult proxy task, we implement it for matrix factorization, and find it achieves faster convergence and lower final loss. Extending this approach to larger-scale tasks, we observe consistent performance gains across domains. We achieve improvements in vision-language tasks and especially notable increases in image generation, with CMMD scores improving by up to 15 points.
BibTeX
@misc{teterwak2024op,
  author = {Piotr Teterwak and Kate Saenko and Bryan A. Plummer and Ser-Nam Lim},
  title = {{OP-LoRA: The Blessing of Dimensionality}},
  year = {2024},
  eprint = {2412.10362},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.10362},
}
Figure