Abstract
Low-Rank Adaptation (LoRA) has become the standard for efficient Large Language Model (LLM) fine-tuning, sparking a trend toward "recycling" or merging these adapters to combine multiple skills. This position paper argues that the current focus on creating new merging and routing algorithms is premature. We demonstrate that existing methods fail to achieve genuine compositional generalization—the ability to link disjoint facts for multi-hop reasoning—unless specific reasoning patterns are explicitly included in the fine-tuning data. Through theoretical analysis and synthetic experiments, we show that simple weight merging often results in shallow pattern matching rather than logical integration. We introduce "bridge" datasets as a critical requirement for unlocking reasoning and urge the community to shift its focus toward understanding the fundamental mechanisms of adapter interaction.
Introduction: The Compositional Gap in LoRA Recycling
The modularity of LoRA has led to a paradigm where adapters are treated as plug-and-play components. However, a critical question remains: can a model composed of separate LoRAs (e.g., one knowing "A is married to B" and another knowing "B lives in London") answer a multi-hop query like "Where does A’s spouse live?"
This paper investigates the limits of this "recycling" approach. We find that without specific intervention, merged models struggle to perform even simple two-hop inferences. This suggests that the community's focus on building new merging heuristics may be overlooking the fundamental architectural constraints of LoRA itself.
Theoretical and Empirical Limits
Our theoretical analysis using simplified single-layer transformers reveals that rank-one updates for individual facts do not naturally compose. Mathematically, summing two LoRAs typically produces a linear mixture of the facts rather than a logically integrated structure.
Empirically, we tested this across various model families, including Qwen and DeepSeek, using synthetic two-hop reasoning tasks.
As illustrated in the figure above, standard libraries combining two or three adapters consistently fail to achieve significant accuracy on inference tasks, with performance often remaining below 10%. This failure persists across model scales, indicating that larger models do not automatically solve the compositional gap through emergent abilities.
The Bridge Technique: Unlocking Multi-Hop Reasoning
To determine what is required to "unlock" reasoning in merged models, we introduced bridge datasets. These datasets do not just provide raw facts; they include Chain-of-Thought (CoT) templates that explicitly model the reasoning path between disjoint facts.
The results are transformative. When LoRAs are trained using these reasoning bridges (Setup 2), accuracy on multi-hop tasks skyrockets. For instance, on the Qwen2.5-14B model, accuracy jumped from negligible levels to 94.1%. This confirms that compositional reasoning is not a byproduct of merging but must be explicitly baked into the training data through reasoning patterns.
Key Insights and Discussion
- Subspace Specificity: LoRAs target very specific low-dimensional subspaces. If a multi-hop query does not align with the subspaces targeted during individual adapter training, the model fails to connect the facts.
- The Familiarity Gap: Models perform significantly better on "Real" data (familiar names/locations) than "Fake" data. This suggests that many perceived successes in LoRA merging are actually the model relying on pre-existing knowledge from its pre-training phase.
- Fragility in Specialized Models: Merging LoRAs into math-specialized models (like Qwen2.5-Math) often degraded performance, as the style of the merged adapters conflicted with the model's internal reasoning logic.
Conclusion: A Call for Rigorous Mechanism Analysis
We advocate for a pause in the "algorithmic arms race" of LoRA merging. Instead of inventing the next merging heuristic, researchers should focus on building rigorous mechanisms to understand when and why these adapters fail to communicate. For practitioners, the message is clear: if you intend to merge LoRAs for complex tasks, you must include "bridge" data that mirrors the target reasoning pattern. Genuine compositional generalization requires more than just weight averaging; it requires explicit logical scaffolding.
Summary updated