MultiLoRA: Democratizing LoRA for Better Multi-Task Learning

Yiming Wang, Yu Lin, Xiaodong Zeng, Guannan Zhang

Abstract

LoRA achieves remarkable resource efficiency and comparable performance when adapting LLMs for specific tasks. Since ChatGPT demonstrated superior performance on various tasks, there has been a growing desire to adapt one model for all tasks. However, the explicit low-rank of LoRA limits the adaptation performance in complex multi-task scenarios. LoRA is dominated by a small number of top singular vectors while fine-tuning decomposes into a set of less important unitary transforms. In this paper, we propose MultiLoRA for better multi-task adaptation by reducing the dominance of top singular vectors observed in LoRA. MultiLoRA scales LoRA modules horizontally and change parameter initialization of adaptation matrices to reduce parameter dependency, thus yields more balanced unitary subspaces. We unprecedentedly construct specialized training data by mixing datasets of instruction follow, natural language understanding, world knowledge, to cover semantically and syntactically different samples. With only 2.5% of additional parameters, MultiLoRA outperforms single LoRA counterparts and fine-tuning on multiple benchmarks and model scales. Further investigation into weight update matrices of MultiLoRA exhibits reduced dependency on top singular vectors and more democratic unitary transform contributions.

Introduction: The Polarization Bottleneck

Low-Rank Adaptation (LoRA) has become a standard technique for Parameter-Efficient Fine-Tuning (PEFT), enabling the adaptation of Large Language Models (LLMs) by training small low-rank matrices. However, standard LoRA faces a significant challenge in multi-task learning: polarization.

Through Singular Value Decomposition (SVD) analysis of the weight update matrices, researchers found that LoRA's updates are dominated by a very small number of top singular vectors. This "top-heavy" distribution contrasts sharply with full fine-tuning, which exhibits a more "democratic" distribution across a larger set of unitary transforms. While polarization is sufficient for single-task adaptation, it lacks the expressive capacity required for complex multi-task scenarios involving logic, world knowledge, and instruction following.

Singular value distribution comparison between LoRA and full fine-tuning. LoRA shows a bimodal, polarized distribution, whereas fine-tuning is more balanced.

MultiLoRA: Horizontal Scaling and Democratization

To bridge the gap between LoRA and full fine-tuning, MultiLoRA introduces three architectural innovations designed to distribute the weight updates more evenly:

  1. Horizontal Scaling: Instead of a single high-rank LoRA module, MultiLoRA utilizes $n$ parallel, independent LoRA modules. This structure reduces parameter dependency and allows the model to capture more diverse feature transforms across independent subspaces.
  2. Kaiming-Uniform Initialization: Unlike standard LoRA, which initializes one matrix to zero, MultiLoRA uses Kaiming-Uniform initialization for up-projection matrices. This expands the initial optimization search space and breaks the symmetry between parallel modules.
  3. Learnable Scaling Factors: To maintain the stability of starting from a pre-trained state, MultiLoRA introduces a zero-initialized learnable scaling factor for each module. This ensures that the initial weight update is zero, allowing the model to gradually integrate new transforms.
The MultiLoRA architecture featuring parallel low-rank modules with independent initialization and learnable scaling factors.

Experimental Evaluation

MultiLoRA was evaluated using a diverse mixture of datasets, including Alpaca (Instruction Following), MMLU (World Knowledge), GSM8K (Arithmetic Reasoning), and SuperGLUE (NLU), across LLaMA model scales from 7B to 65B.

Key Performance Gains

  • Superior Multi-Task Performance: On the MMLU benchmark, MultiLoRA (LLaMA-7B) outperformed standard LoRA by 3.5%. On the RTE task, it achieved a 5.9% improvement.
  • Competitive with Fine-Tuning: MultiLoRA frequently outperformed full fine-tuning on 7B and 13B models while using significantly fewer trainable parameters (approx. 2.5% additional parameters).
  • Robustness: MultiLoRA showed much lower performance variance across different tasks compared to standard LoRA, indicating better generalization.

Efficiency and Throughput

MultiLoRA retains the zero-inference overhead of standard LoRA, as the parallel modules can be merged into the base model weights. In terms of training efficiency, it offers nearly double the throughput of full fine-tuning, although VRAM usage increases with the number of parallel modules due to activation caching.

Conclusion

MultiLoRA addresses the expressive limitations of standard LoRA in multi-task environments by "democratizing" the contribution of singular vectors. By scaling modules horizontally and optimizing initialization, MultiLoRA achieves a more balanced update distribution similar to full fine-tuning. This approach provides a powerful and efficient path for creating versatile, multi-task LLMs without the prohibitive costs of full-parameter updates.

Summary updated

Abstract

LoRA achieves remarkable resource efficiency and comparable performance when adapting LLMs for specific tasks. Since ChatGPT demonstrated superior performance on various tasks, there has been a growing desire to adapt one model for all tasks. However, the explicit low-rank of LoRA limits the adaptation performance in complex multi-task scenarios. LoRA is dominated by a small number of top singular vectors while fine-tuning decomposes into a set of less important unitary transforms. In this paper, we propose MultiLoRA for better multi-task adaptation by reducing the dominance of top singular vectors observed in LoRA. MultiLoRA scales LoRA modules horizontally and change parameter initialization of adaptation matrices to reduce parameter dependency, thus yields more balanced unitary subspaces. We unprecedentedly construct specialized training data by mixing datasets of instruction follow, natural language understanding, world knowledge, to cover semantically and syntactically different samples. With only 2.5% of additional parameters, MultiLoRA outperforms single LoRA counterparts and fine-tuning on multiple benchmarks and model scales. Further investigation into weight update matrices of MultiLoRA exhibits reduced dependency on top singular vectors and more democratic unitary transform contributions.
BibTeX
@misc{wang2023multilora,
  author = {Yiming Wang and Yu Lin and Xiaodong Zeng and Guannan Zhang},
  title = {{MultiLoRA: Democratizing LoRA for Better Multi-Task Learning}},
  year = {2023},
  eprint = {2311.11501},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.11501},
}
Figure