When deep neural networks are fine-tuned on poisoned datasets, they acquire "backdoors"—hidden conditional behaviors that trigger malicious outputs only when a specific, arbitrary token or pattern is present. Traditionally, removing these backdoors required expensive retraining or access to the original clean training data.
However, a wave of recent research has uncovered a surprising geometric property of neural network parameter spaces: model merging naturally dilutes and neutralizes backdoor signals. By mathematically blending the weights of a compromised model with one or more benign models, the idiosyncratic weight clusters responsible for the backdoor are washed out, while the shared, robust representations responsible for general task performance are preserved.
The Intuition: Shared Manifolds vs. Idiosyncratic Outliers
To understand why model merging acts as an effective defense, we must look at how task capabilities and backdoors are structurally represented within a model's parameter space.
When multiple models are trained on the same task (even on different datasets), they converge toward a shared functional manifold. The core parameters required to understand language, follow instructions, or classify images are dense, highly correlated, and distributed consistently across models.
Backdoors, by contrast, are highly localized anomalies. As Terry Tong et al. observe, malicious triggers typically manifest as sparse "outlier weights"—narrow parameter adjustments tailored to a specific, artificial poisoning pattern. Because these updates are distinct from general task features, they occupy a completely different, non-overlapping direction in the parameter space.
Model A (Backdoored): [=== Task Features ===] + [*** Backdoor Outlier ***]
Model B (Clean): [=== Task Features ===] + [ No Backdoor ]
----------------------------------------------------------------------------
Merged (Averaged): [=== Task Features ===] + [ * Half-Strength Signal * ]
When you average the weights of a backdoored model with a clean model, the shared task features undergo constructive interference, reinforcing the model's general utility. However, the unique backdoor weights undergo destructive interference or dilution. The malicious signal is effectively halved or reduced to a minority fraction, dropping it below the activation threshold required to trigger the behavior at inference time.
Paradigms of Weight-Space Sanitization
This dilution effect can be leveraged as a defense mechanism through several distinct merging strategies, depending on what resources are available.
1. Mutual Sanitization (No Clean Reference Model)
Remarkably, you do not even need a guaranteed "clean" model to purge backdoors. Ansh Arora et al. demonstrate a phenomenon called mutual model sanitization. If a developer downloads multiple open-source models that are potentially insecure, merging them together will still neutralize individual vulnerabilities. Because the models were trained independently, their core task capabilities align, but their poisoning patterns do not. As long as they do not share the exact same backdoor trigger, the unique malicious weights of each individual model are washed out by the others during weight averaging.
2. Reference and Adapter Blending
When a clean reference model is available, parameter-space merging provides a low-cost security layer without requiring retraining. Qin Liu et al. note that combining a compromised model with a clean base model successfully dilutes malicious associations due to the inherent sparsity of backdoor updates. This applies directly to Parameter-Efficient Fine-Tuning (PEFT) as well; Zhen Sun et al. show that mixing the parameters of a suspicious LoRA adapter with the original clean pre-trained weights significantly drops the attack success rate while preserving the adapter's intended utility.
3. Self-Fusion and Parametric Conflict
If an engineer only possesses a single, compromised model and no external reference, they can force the model to sanitize itself. Weilin Lin et al. introduce a data-free defense that prunes the backdoored model to create a lower-performance, backdoor-free variant. Fusing this pruned model back into the original high-performance model allows the clean structural features to overwrite the malicious ones.
Taking a different approach to self-sanitization, Chen Chen et al. intentionally introduce "information conflicts." By training a small adapter on a tiny fraction of clean data and merging it into the compromised model, they embed contradictory behavioral patterns. This weight-level integration creates a parametric conflict that disrupts the hidden trigger mechanism entirely without needing to detect or know what the trigger looks like.
The Adversarial Arms Race: Surviving Dilution
Because standard model merging algorithms (like simple weight averaging or sparsification) naturally neutralize malicious triggers, adversaries can no longer rely on traditional poisoning attacks if their models are intended to be used in collaborative open-source ecosystems. This has forced attackers to develop merging-aware exploits.
To counteract the dilution effect, Zenghui Yuan et al. propose an attack called "Merge Hijacking." Recognizing that merging will scale down their malicious parameter adjustments, the authors explicitly amplify and rescale the malicious parameter vector before uploading the model. By artificially inflating the magnitude of the backdoor weights, they ensure that even after the signal is diluted by a merging algorithm, the remaining vector is still strong enough to cross the activation threshold and trigger the backdoor.
Similarly, Jianfeng Chi et al. developed "BadMerging," an attack framework mathematically optimized to survive blending operations. BadMerging ensures that the trigger-response mechanism remains active even when the compromised model is assigned a very low blending coefficient by the developer.
Ultimately, the effectiveness of model merging as a security tool stems from a fundamental asymmetry: valid task features are structurally robust because they are learned from real-world distributions, whereas backdoors are fragile, artificial constructs. Unless an attacker explicitly hardens their weights against parameter blending, the simple mathematical act of averaging models remains one of the most effective, zero-shot defenses against neural deployment threats.
Key Foundational Work
Here's a Free Lunch: Sanitizing Backdoored Models with Model Merge (2024) — contributed 7 concept elements to this direction.
BadJudge: Backdoor Vulnerabilities of LLM-As-A-Judge (2025) — contributed 2 concept elements to this direction.
Neutralizing Backdoors through Information Conflicts for Large Language Models (2024) — contributed 2 concept elements to this direction.