Vulnerabilities and Defenses in Model Merging

35 concepts15 papers1 proposal

Model merging has emerged as a powerful paradigm for creating multi-task neural networks without the prohibitive cost of retraining. By treating the weights of models fine-tuned from a common checkpoint as vectors in a shared parameter space, developers can use simple algebraic operations—like weight averaging or task arithmetic—to combine specialized capabilities into a single unified model.

However, this elegant assumption of linearity introduces a massive security and intellectual property (IP) blind spot. Because models are typically evaluated individually before they are combined, traditional pre-merge security checks fail to anticipate how parameters will interact once blended. This creates a unique attack surface where malicious actors can inject hidden behaviors, steal data, or bypass licensing restrictions.

The Attack Surface of Parameter Blending

The core intuition behind model merging attacks is that weight-space arithmetic allows malicious signals to bypass standard safety filters. When an adversary uploads a poisoned model to an open-source repository, they exploit the fact that the merging process is designed to preserve and combine parameter-level features.

Backdoor Succession and Amplification

Why do backdoors survive weight averaging instead of being diluted? In task arithmetic, a backdoor trigger can be represented as a sparse parameter vector that occupies a orthogonal region to the primary task features. When merged, these triggers persist into the final multi-task architecture.

Wang et al. demonstrate that upstream models can appear completely benign and perform flawlessly on standard benchmarks individually, yet harbor hidden functionalities that only activate post-merge. This supply chain risk means a single malicious contributor can hijack a merged system, a vulnerability explored by both Chi et al. and Yuan et al.. Counter-intuitively, the merging process can even concentrate and amplify these malicious signals rather than washing them out, as formalized by Pawlak et al. through the lens of "Backdoor Vectors."

[Clean Model A] \
                 +---> [Merged Model] ---> (Backdoor Activates!)
[Benign-Looking] /
[Poisoned Model]

Parameter-Level Data Theft

Merging can also be weaponized to extract sensitive information from proprietary models. If an attacker wants to steal Personally Identifiable Information (PII) or private training data embedded within a target model, they can design a "phishing" or "stealer" model disguised as a harmless task expert.

When a user merges this phishing model with their aligned target model, the malicious parameters degrade the target's safety alignment. Guo et al. and Lu et al. show that this parameter blending causes the merged model to lose its security awareness, forcing it to disclose sensitive data that its original aligned counterpart would have refused to output. Even privacy-preserving variants like partial model merging—where only specific layers are shared—fail to protect data; Wu et al. show that shared parameters contain enough internal processing information to reconstruct the functionality of the unshared, private layers.

The IP Dilemma and Fingerprint Evasion

Beyond direct security threats, model merging provides a low-cost mechanism for bypassing intellectual property protections. When weights are averaged, traditional weight-based watermarks or quantization signatures are easily obfuscated or stripped away, allowing third parties to create derivative models that retain proprietary utility while erasing ownership signatures.

To counter this, researchers have shifted from static parameter watermarks to robust, merge-resistant fingerprinting. Cong et al. demonstrate that while weight-level watermarks fail, behavioral or instructional fingerprinting remains highly resilient to weight blending.

To make black-box verification survive extensive merging, Waseda et al. and the authors of MERGEPRINT propose optimizing model weights against a simulated "pseudo-merged" environment during the initial training phase. By anticipating the weight-averaging process mathematically, they ensure that the ownership signature remains retrievable even after the model is blended with multiple other architectures.

Proactive Defense: Making Models Unmergeable

If unauthorized model merging threatens governance, safety, and IP, how can developers prevent it entirely? The intuition behind "unmergeable" models is to break the linear compatibility of the parameter space without degrading standalone performance.

Standard Model:  [Weights] -------------------------> Merges cleanly
Unmergeable:     [Weights + Secret Transform] ------> Destructive Interference

One effective strategy exploits structural symmetry within network layers. In the MergeLock framework introduced by Wang et al., secret, invertible transformations are applied to the parameters of self-attention layers. Because the internal transformations are perfectly inverted during standard inference, the model functions normally on its own. However, if an unauthorized user attempts to average these weights with another model, the mathematical alignment breaks down, causing catastrophic interference and a near-total loss of accuracy.

Another approach targets the model's loss landscape. Jang et al. introduce TRAP², a training-time defense that shapes the loss landscape to be hypersensitive to weight re-scaling. Similarly, the proactive defense framework PaRaMS (Disrupting Model Merging) applies subtle parameter perturbations. In both cases, the standalone model rests securely in a stable local minimum, but any parameter-level blending pushes the weights into a chaotic, non-functional region of the loss landscape, rendering the unauthorized merge useless.

Reactive Defense: Filtering Malice During Fusion

When preventing a merge is not an option, developers must find ways to safely fuse models from untrusted or public sources. This requires defense mechanisms that can distinguish between beneficial task knowledge and harmful malicious injections within the parameter space.

Yang et al. address this with Defense-Aware Merging (DAM), which employs a dual-mask optimization strategy. DAM identifies a "safety-aware subspace" within the parameters, effectively filtering out the specific parameter regions responsible for backdoor triggers while preserving multi-task accuracy.

Shifting the defensive battleground from raw parameters to the feature space, Zhu et al. introduce Linear Feature Path Minimization (LFPM). By leveraging the linear behavior of features across tasks rather than just weight values, LFPM neutralizes the propagation of malicious triggers, effectively vaccinating the unified backbone against supply chain attacks without sacrificing performance.

Key Foundational Work

From Purity to Peril: Backdooring Merged Models From "Harmless" Benign Components (2025) — contributed 8 concept elements to this direction.
Merger-as-a-Stealer: Stealing Targeted PII from Aligned LLMs with Model Merging (2025) — contributed 7 concept elements to this direction.
Mitigating the Backdoor Effect for Multi-Task Model Merging via Safety-Aware Subspace (2025) — contributed 5 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_model_merging_vulnerability_8aabcb1d,
  title = {model merging vulnerability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/model-merging-vulnerability/rdr_8aabcb1dee4a40b6a72df020d6148b10},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_model_merging_vulnerability_8aabcb1d_v1,
  title = {model merging vulnerability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/model-merging-vulnerability/rdr_8aabcb1dee4a40b6a72df020d6148b10/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake