Position: Pause Recycling LoRAs and Prioritize Mechanisms to Uncover Limits and Effectiveness

Mei-Yen Chen, Thi Thu Uyen Hoang, Michael Hahn, M. Saquib SarfrazarXiv 2025 · DOI · arXiv

Abstract

Low-Rank Adaptation (LoRA) has become the standard for efficient Large Language Model (LLM) fine-tuning, sparking a trend toward "recycling" or merging these adapters to combine multiple skills. This position paper argues that the current focus on creating new merging and routing algorithms is premature. We demonstrate that existing methods fail to achieve genuine compositional generalization—the ability to link disjoint facts for multi-hop reasoning—unless specific reasoning patterns are explicitly included in the fine-tuning data. Through theoretical analysis and synthetic experiments, we show that simple weight merging often results in shallow pattern matching rather than logical integration. We introduce "bridge" datasets as a critical requirement for unlocking reasoning and urge the community to shift its focus toward understanding the fundamental mechanisms of adapter interaction.

Introduction: The Compositional Gap in LoRA Recycling

The modularity of LoRA has led to a paradigm where adapters are treated as plug-and-play components. However, a critical question remains: can a model composed of separate LoRAs (e.g., one knowing "A is married to B" and another knowing "B lives in London") answer a multi-hop query like "Where does A’s spouse live?"

This paper investigates the limits of this "recycling" approach. We find that without specific intervention, merged models struggle to perform even simple two-hop inferences. This suggests that the community's focus on building new merging heuristics may be overlooking the fundamental architectural constraints of LoRA itself.

Theoretical and Empirical Limits

Our theoretical analysis using simplified single-layer transformers reveals that rank-one updates for individual facts do not naturally compose. Mathematically, summing two LoRAs typically produces a linear mixture of the facts rather than a logically integrated structure.

Empirically, we tested this across various model families, including Qwen and DeepSeek, using synthetic two-hop reasoning tasks.

Accuracy of merged LoRA libraries across different model scales, showing the failure of simple combinations to achieve multi-hop reasoning.

As illustrated in the figure above, standard libraries combining two or three adapters consistently fail to achieve significant accuracy on inference tasks, with performance often remaining below 10%. This failure persists across model scales, indicating that larger models do not automatically solve the compositional gap through emergent abilities.

The Bridge Technique: Unlocking Multi-Hop Reasoning

To determine what is required to "unlock" reasoning in merged models, we introduced bridge datasets. These datasets do not just provide raw facts; they include Chain-of-Thought (CoT) templates that explicitly model the reasoning path between disjoint facts.

The Bridge technique: incorporating reasoning templates (Setup 2) significantly improves compositional generalization compared to raw fact merging (Setup 1).

The results are transformative. When LoRAs are trained using these reasoning bridges (Setup 2), accuracy on multi-hop tasks skyrockets. For instance, on the Qwen2.5-14B model, accuracy jumped from negligible levels to 94.1%. This confirms that compositional reasoning is not a byproduct of merging but must be explicitly baked into the training data through reasoning patterns.

Key Insights and Discussion

  • Subspace Specificity: LoRAs target very specific low-dimensional subspaces. If a multi-hop query does not align with the subspaces targeted during individual adapter training, the model fails to connect the facts.
  • The Familiarity Gap: Models perform significantly better on "Real" data (familiar names/locations) than "Fake" data. This suggests that many perceived successes in LoRA merging are actually the model relying on pre-existing knowledge from its pre-training phase.
  • Fragility in Specialized Models: Merging LoRAs into math-specialized models (like Qwen2.5-Math) often degraded performance, as the style of the merged adapters conflicted with the model's internal reasoning logic.

Conclusion: A Call for Rigorous Mechanism Analysis

We advocate for a pause in the "algorithmic arms race" of LoRA merging. Instead of inventing the next merging heuristic, researchers should focus on building rigorous mechanisms to understand when and why these adapters fail to communicate. For practitioners, the message is clear: if you intend to merge LoRAs for complex tasks, you must include "bridge" data that mirrors the target reasoning pattern. Genuine compositional generalization requires more than just weight averaging; it requires explicit logical scaffolding.

Summary updated

Abstract

Merging or routing low-rank adapters (LoRAs) has emerged as a popular solution for enhancing large language models, particularly when data access is restricted by regulatory or domain-specific constraints. This position paper argues that the research community should shift its focus from developing new merging or routing algorithms to understanding the conditions under which reusing LoRAs is truly effective. Through theoretical analysis and synthetic two-hop reasoning and math word-problem tasks, we examine whether reusing LoRAs enables genuine compositional generalization or merely reflects shallow pattern matching. Evaluating two data-agnostic methods--parameter averaging and dynamic adapter selection--we found that reusing LoRAs often fails to logically integrate knowledge across disjoint fine-tuning datasets, especially when such knowledge is underrepresented during pretraining. Our empirical results, supported by theoretical insights into LoRA's limited expressiveness, highlight the preconditions and constraints of reusing them for unseen tasks and cast doubt on its feasibility as a truly data-free approach. We advocate for pausing the pursuit of novel methods for recycling LoRAs and emphasize the need for rigorous mechanisms to guide future academic research in adapter-based model merging and practical system designs for practitioners.
BibTeX
@misc{chen2025position,
  author = {Mei-Yen Chen and Thi Thu Uyen Hoang and Michael Hahn and M. Saquib Sarfraz},
  title = {{Position: Pause Recycling LoRAs and Prioritize Mechanisms to Uncover Limits and Effectiveness}},
  year = {2025},
  eprint = {2506.13479},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.13479},
}
Figure