Model Unmerging: Making Your Models Unmergeable for Secure Model Sharing

Zihao Wang, Enneng Yang, Lu Yin, Shiwei Liu, Li Shen

Abstract

Model merging leverages multiple finetuned expert models to construct a multi-task model with low cost, and is gaining increasing attention. However, as a growing number of finetuned models become publicly available, concerns about the safety of model merging have emerged. Unauthorized merging may infringe on developers' rights and risk leaking sensitive personal information. Most existing methods focus on detecting whether a merged model originates from a specific source model, but fail to effectively prevent illegal merging. In this paper, we propose MergeLock, an active protection mechanism that disrupts model parameters to render them unmergeable, thereby directly preventing unauthorized model merging. Specifically, leveraging the inherent symmetry of the attention mechanism in Transformer-based models, we randomly sample two pairs of invertible matrices and apply them to the Query-Key (QK) and Value-Output (VO) branches. This transformation keeps the model's output unchanged while pushing it away from the shared parameter space of other finetuned models. Extensive experiments across both vision and language tasks demonstrate that MergeLock can degrade the performance of merged models by over 95% when a protected model is involved in most cases, demonstrating its effectiveness. Moreover, we further demonstrate that merged models protected by MergeLock cannot be effectively recovered using low-cost restoration methods, further enhancing robustness against unauthorized merging.

Introduction

Model merging has emerged as a popular, low-cost technique for creating multi-task models by averaging the weights of several task-specific "expert" models. However, this convenience introduces significant security and intellectual property (IP) risks. Since most fine-tuned models derived from the same pretrained checkpoint (e.g., CLIP or Llama) inhabit the same "loss basin," they are easily combined without the original developer's consent. This leads to IP infringement, potential privacy leaks of training data, and safety bypasses where malicious actors merge safe models with harmful ones.

The threat model of unauthorized model merging and how MergeLock acts as a proactive defense.

Unlike reactive defenses like watermarking, which only detect unauthorized use after the fact, we propose MergeLock. MergeLock is a proactive mechanism that renders Transformer-based models "unmergeable" by breaking the parameter-space compatibility that merging relies on. By applying secret, invertible transformations to model parameters, developers can share high-performing models that function perfectly on their own but fail catastrophically if an unauthorized user attempts to merge them with other models.

Methodology

MergeLock exploits the mathematical symmetry inherent in the Self-Attention layers of Transformers. In a standard attention head, the Query (Q) and Key (K) matrices interact, as do the Value (V) and Output (O) matrices. Because these operations are linear before the softmax function, we can insert secret, invertible matrices to transform the parameters without altering the model's final output.

Specifically, the transformation is defined as:

  • QK Branch: $W_Q' = A^\top W_Q$ and $W_K' = A^{-1} W_K$
  • VO Branch: $W_V' = B^\top W_V$ and $W_O' = W_O B^{-1}$
MergeLock applies invertible transformations to attention weights, shifting the model to a different loss basin while maintaining identical output.

While the matrices $A$ and $B$ cancel out during computation, they push the actual parameter values into a different region of the loss landscape. This isolates the protected model, creating a massive "loss barrier" between it and any other model. An unauthorized attempt to merge these "locked" weights results in a model that falls into a high-loss region, rendering it useless.

This approach is significantly more robust than previous attempts like PaRaMS, which targeted Feedforward Networks (MLPs). MLPs contain non-linear activation functions (like ReLU) that restrict possible transformations to simple permutations. In contrast, the linearity of self-attention allows for continuous transformations such as rotations and scaling, which are much more difficult for an attacker to reverse-engineer.

Experimental Evaluation

We evaluated MergeLock across diverse vision tasks (using CLIP-ViT and ViT-L) and language tasks (using the GLUE benchmark with Flan-T5).

  • Performance Collapse: Merging a MergeLock-protected model typically results in a performance drop of over 95%, often reducing accuracy to near-random levels. In vision tasks, accuracy on datasets like SUN397 or EuroSAT dropped to near-random levels when a protected model was involved in the merge.
  • Robustness to Alignment Attacks: Adversaries attempting to "re-align" parameters using optimization techniques (like the Kabsch algorithm) fail, recovering only about 5% of original performance. This confirms that MergeLock creates a genuine "unmergeability" property that is not easily bypassed by low-cost restoration methods.
  • Scalability and Invariance: The protection remains effective across different architectures and model sizes, from ViT-Base to ViT-Large, and proved equally effective for both vision Transformers and language models (Flan-T5).

Discussion and Conclusion

MergeLock provides a robust solution for secure model sharing by leveraging the structural properties of Transformers. By breaking the "Linear Mode Connectivity" that successful merging depends on, it effectively isolates protected models in the parameter space. While currently optimized for attention mechanisms, future work will explore extending these principles to other architectures like CNNs or State Space Models. Additionally, validating MergeLock on massive-scale LLMs remains a key direction for future research.

Summary updated

Abstract

Model merging leverages multiple finetuned expert models to construct a multi-task model with low cost, and is gaining increasing attention. However, as a growing number of finetuned models become publicly available, concerns about the safety of model merging have emerged. Unauthorized merging may infringe on developers' rights and risk leaking sensitive personal information. Most existing methods focus on detecting whether a merged model originates from a specific source model, but fail to effectively prevent illegal merging. In this paper, we propose MergeLock, an active protection mechanism that disrupts model parameters to render them unmergeable, thereby directly preventing unauthorized model merging. Specifically, leveraging the inherent symmetry of the attention mechanism in Transformer-based models, we randomly sample two pairs of invertible matrices and apply them to the Query-Key (QK) and Value-Output (VO) branches. This transformation keeps the model's output unchanged while pushing it away from the shared parameter space of other finetuned models. Extensive experiments across both vision and language tasks demonstrate that MergeLock can degrade the performance of merged models by over 95% when a protected model is involved in most cases, demonstrating its effectiveness. Moreover, we further demonstrate that merged models protected by MergeLock cannot be effectively recovered using low-cost restoration methods, further enhancing robustness against unauthorized merging. The code is available at https://github.com/hetailang/Merge-Lock.
BibTeX
@misc{wang2025model,
  author = {Zihao Wang and Enneng Yang and Lu Yin and Shiwei Liu and Li Shen},
  title = {{Model Unmerging: Making Your Models Unmergeable for Secure Model Sharing}},
  year = {2025},
  eprint = {2509.01548},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.01548},
}
Figure