ShareLoRA: Less Tuning, More Performance for LoRA Fine-tuning of LLMs
Its authors withdrew this paper, so there is no full text to read or summarize.
Lists
Only you see this
Abstract
Fine-tuning large language models (LLMs) is prohibitively expensive, prompting the development of various parameter-efficient fine-tuning (PEFT) methods. These methods primarily focus on fine-tuning small, additional modules known as adapters, which account for only a small fraction of the total LLM parameters. One such method, low-rank adaptation (LoRA), has shown notable parameter efficiency while maintaining performance comparable to full fine-tuning. However, classical LoRA may still involve tuning more parameters than necessary given the intrinsic rank of pre-trained weights, as highlighted by prior work. In this work, we introduce ShareLoRA, a novel approach that further enhances parameter efficiency during LLM fine-tuning by leveraging redundancies in pre-trained model weights to share LoRA modules, thereby significantly reducing the number of trainable parameters. Specifically, ShareLoRA automatically identifies redundancies in the pre-trained weights and determines which LoRA adapters can share parameters. This is achieved by measuring the similarity between representations to assess information redundancy and using a greedy algorithm to maximize parameter sharing. We conducted extensive evaluations on the LLMs of the LLaMA family across benchmark tasks. Notably, ShareLoRA achieves better parameter efficiency, with up to a 23\% reduction in the number of fine-tuned parameters while delivering performance comparable to or better than existing PEFT methods.
BibTeX
@misc{shen2025sharelora,
author = {Zheyu Shen and Guoheng Sun and Yexiao He and Ziyao Wang and Yuning Zhang and Souvik Kundu and Eric P. Xing and Hongyi Wang and Ang Li},
title = {{ShareLoRA: Less Tuning, More Performance for LoRA Fine-tuning of LLMs}},
year = {2025},
url = {https://openreview.net/forum?id=O6QZ4W6GXt},
howpublished = {OpenReview},
note = {Submitted to ICLR 2025},
}Loading related content...