ShareLoRA: Less Tuning, More Performance for LoRA Fine-tuning of LLMs

Zheyu Shen, Guoheng Sun, Yexiao He, Ziyao Wang, Yuning Zhang, Souvik Kundu, Eric P. Xing, Hongyi Wang, Ang LiOpenReview 2025 · Submission to ICLR 2025 · Withdrawn · OpenReview

Its authors withdrew this paper, so there is no full text to read or summarize.

Abstract

Fine-tuning large language models (LLMs) is prohibitively expensive, prompting the development of various parameter-efficient fine-tuning (PEFT) methods. These methods primarily focus on fine-tuning small, additional modules known as adapters, which account for only a small fraction of the total LLM parameters. One such method, low-rank adaptation (LoRA), has shown notable parameter efficiency while maintaining performance comparable to full fine-tuning. However, classical LoRA may still involve tuning more parameters than necessary given the intrinsic rank of pre-trained weights, as highlighted by prior work. In this work, we introduce ShareLoRA, a novel approach that further enhances parameter efficiency during LLM fine-tuning by leveraging redundancies in pre-trained model weights to share LoRA modules, thereby significantly reducing the number of trainable parameters. Specifically, ShareLoRA automatically identifies redundancies in the pre-trained weights and determines which LoRA adapters can share parameters. This is achieved by measuring the similarity between representations to assess information redundancy and using a greedy algorithm to maximize parameter sharing. We conducted extensive evaluations on the LLMs of the LLaMA family across benchmark tasks. Notably, ShareLoRA achieves better parameter efficiency, with up to a 23\% reduction in the number of fine-tuned parameters while delivering performance comparable to or better than existing PEFT methods.
BibTeX
@misc{shen2025sharelora,
  author = {Zheyu Shen and Guoheng Sun and Yexiao He and Ziyao Wang and Yuning Zhang and Souvik Kundu and Eric P. Xing and Hongyi Wang and Ang Li},
  title = {{ShareLoRA: Less Tuning, More Performance for LoRA Fine-tuning of LLMs}},
  year = {2025},
  url = {https://openreview.net/forum?id=O6QZ4W6GXt},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2025},
}

Loading related content...

Figure