Abstract
Low-Rank Adaptation (LoRA) has become the standard for parameter-efficient fine-tuning of Large Language Models (LLMs). However, storing unique adapters for every layer remains a bottleneck for large-scale deployments. We present Tied-LoRA, a novel approach that ties weights across model layers to drastically reduce parameter counts. Our findings show that the TL6 configuration achieves performance parity with standard LoRA while utilizing as little as 12.5% of the trainable parameters.
Introduction
Large Language Models (LLMs) are increasingly customized for specific tasks using Low-Rank Adaptation (LoRA). While LoRA is significantly more efficient than full-parameter fine-tuning, it still requires a unique pair of low-rank matrices ($A$ and $B$) for every transformer layer. For service providers managing thousands of task-specific adapters, this creates substantial storage and operational overhead. Tied-LoRA addresses this scaling problem by sharing (tying) these matrices across the entire model architecture.
Methods
Tied-LoRA introduces a flexible framework for weight tying within the LoRA paradigm. The architecture augments the standard low-rank projection with layer-specific scaling vectors, $u$ and $v$, represented as diagonal matrices.
The researchers explored six configurations (TL1–TL6) by varying which components are tied, frozen, or trainable. The most effective configuration, TL6 ($vB_\infty uA_\infty$), ties the core matrices $A$ and $B$ across all layers while keeping them trainable. To maintain layer-specific expressivity, the scaling vectors $u$ and $v$ remain unique to each layer and are also trainable. This allows the model to learn a global structural update while permitting each layer to individually modulate that update's impact.
Results
Tied-LoRA was evaluated across extractive QA (SQuAD), summarization (DialogSum), commonsense reasoning (HellaSwag), translation (IWSLT), and mathematical reasoning (GSM8K) using LLaMA-2 7B and GPT-2B-001 base models.
Key findings include:
- Parameter Efficiency: TL6 reduces the trainable parameter count to approximately 12.5%–15% of standard LoRA. For a LLaMA-2 7B model at rank 8, this translates to a reduction from ~4.2M to ~530K parameters.
- Performance Parity: Across most tasks, TL6 performed within 1–2% of standard LoRA. Notably, in German-to-English translation on LLaMA-2 7B, TL6 outperformed standard LoRA despite the massive parameter reduction.
- Rank Stability: Unlike other tied-weight methods that degrade at higher ranks, TL6 remains stable up to rank 128.
- Global vs. Local: The success of TL6 suggests that transformer layers often benefit from similar structural updates, which can be captured by a single global adapter.
Conclusion
Tied-LoRA, particularly the TL6 configuration, offers a compelling solution for ultra-efficient LLM adaptation. By leveraging global updates with layer-specific scaling, it maintains high performance while significantly lowering storage requirements. Future work will explore the scalability of this method to models exceeding 70B parameters and investigate closing the performance gap observed in complex mathematical reasoning tasks.
Summary updated