Will it Merge? On The Causes of Model Mergeability

Adir Rahamim, Asaf Yehudai, Boaz Carmeli, Leshem Choshen, Yosi Mass, Yonatan Belinkov

TL;DR

This paper introduces "mergeability," a quantifiable metric for how well fine-tuned model updates integrate into a larger model. They discover that a model's existing knowledge is the primary driver of mergeability, with models fine-tuned on data they already understand merging much better. This insight leads to a novel weighted merging strategy that prioritizes less-understood tasks.

What Problem Does This Paper Address?

Large pre-trained language models are often fine-tuned for specific tasks. A promising technique called "model merging" aims to combine multiple such fine-tuned models into a single, versatile model without retraining. However, it's been unclear why some fine-tuned models merge seamlessly while others fail, leading to performance degradation. This lack of understanding hinders the development of more robust merging strategies and the creation of truly mergeable models. This paper tackles this by defining and investigating the concept of "mergeability" – a property of model updates that dictates their success when combined.

Methods

The core of this work is the introduction and empirical validation of a "mergeability score." This score quantifies how well a specific model update (in this case, LoRA adapters) retains its performance when merged with a random subset of other model updates. They then analyze various factors that might influence this score, including:

  • Base model knowledge: How well the original pre-trained model understood the data the fine-tuned model was trained on.
  • Weight properties: Structural characteristics of the model's learned weights.
  • Training data difficulty: Proxies like perplexity and context length of the fine-tuning data.

They also explore whether mergeability is an intrinsic property of an update or dependent on the merging set. Finally, they propose a weighted merging technique based on their findings.

Datasets

  • PopQA: Used for example-level mergeability analysis. This is an open-domain, entity-centric question-answering benchmark.
  • Lots-of-LoRAs: Used for task-level mergeability analysis. This is a large collection of LoRA adapters trained on diverse NLP tasks.

Key Insights and Ideas

  • Mergeability is a real, measurable property: The paper demonstrates that not all model updates are created equal when it comes to merging. There's a spectrum of "mergeability," and it's not just random chance.
  • Base model knowledge is king: The most significant factor influencing mergeability is how well the original pre-trained model understood the data used for fine-tuning. If the base model already "knew" the answer or concept, the fine-tuned update is much more likely to merge well.
  • Mergeability is local: An update's mergeability is largely an intrinsic property of the update itself, rather than being heavily dependent on the specific set of other updates it's merged with.
  • "Easier" tasks merge better: Examples that the base model already performed well on (low probability gap between top and correct answers) are more mergeable. Conversely, tasks where the base model struggled are less mergeable.
  • A new merging strategy: By understanding that less mergeable tasks (those the base model struggled with) are often more critical to preserve, the authors propose a weighted averaging approach that gives more importance to these tasks.

Key Results

  • Strong correlation between base model knowledge and mergeability: In PopQA, a smaller gap between the base model's predicted probability for the correct answer and the highest probability answer correlates with higher mergeability. Similarly, in Lots-of-LoRAs, tasks with higher base model accuracy exhibit better mergeability.
  • Weight properties and data difficulty are weak predictors: The study found little to no consistent correlation between mergeability and metrics like weight norms, singular values, or training data perplexity/length.
  • Mergeability is intrinsic: Experiments showed that a highly mergeable update remained stable when merged with different groups of other updates, supporting its intrinsic nature.
  • Weighted merging improves performance: The proposed weighted averaging strategy, which prioritizes tasks with lower base model accuracy, successfully retained performance on low-accuracy tasks while minimizing degradation on high-accuracy tasks.

Limitations and Future Work

  • LoRA focus: The primary experiments used LoRA adapters. While the authors suggest extension to other fine-tuning methods, further validation would be beneficial.
  • Merging algorithms: While KNOTS was the primary algorithm, further exploration of other merging techniques like TIES and simple averaging is suggested.
  • Dataset diversity: The core findings were validated on PopQA and Lots-of-LoRAs. Extending to more diverse datasets and tasks would strengthen the generalizability of the results.
  • Malicious use: The authors acknowledge that understanding mergeability could potentially be exploited by malicious actors to amplify unwanted behaviors in merged models.

Summary updated

Abstract

Model merging has emerged as a promising technique for combining multiple fine-tuned models into a single multitask model without retraining. However, the factors that determine whether merging will succeed or fail remain poorly understood. In this work, we investigate why specific models are merged better than others. To do so, we propose a concrete, measurable definition of mergeability. We investigate several potential causes for high or low mergeability, highlighting the base model knowledge as a dominant factor: Models fine-tuned on instances that the base model knows better are more mergeable than models fine-tuned on instances that the base model struggles with. Based on our mergeability definition, we explore a simple weighted merging technique that better preserves weak knowledge in the base model.
BibTeX
@inproceedings{rahamim2026will,
  author = {Adir Rahamim and Asaf Yehudai and Boaz Carmeli and Leshem Choshen and Yosi Mass and Yonatan Belinkov},
  title = {{Will it Merge? On The Causes of Model Mergeability}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1322},
}

Loading related content...

Figure