TL;DR
Retraining deepfake detectors every time a new generative model emerges is computationally prohibitive, while multi-task joint training often suffers from negative task interference. Real-aware Residual Model Merging ($R^2M$) introduces a training-free parameter-space merging technique that disentangles a shared "Real" representation from generator-specific "Fake" residuals using low-rank factorization. By retaining a single core Real component and aggregating denoised, norm-matched Fake residuals, $R^2M$ matches or beats joint training on seen forgeries and significantly improves zero-shot transfer to unseen deepfake generators.
What Problem Does This Paper Address?
Generative video and image synthesis (diffusion models, talking heads, advanced face-swapping) evolves at a breakneck pace. Defending against deepfakes usually faces an operational dilemma:
- Full Retraining / Joint Training: Gathering data from every new generator and retraining an "all-in-one" detector is slow, costly, and vulnerable to catastrophic forgetting or cross-task interference.
- Specialist Proliferation: Training independent specialist detectors for each generator family creates an unmanageable ensemble footprint in production.
Model merging—combining fine-tuned weights directly in parameter space without gradient descent—offers an appealing middle ground. However, standard merging algorithms (e.g., simple weight averaging, Task Arithmetic, TIES) were built for multi-task problems with disjoint classification labels. In contrast, deepfake detection is inherently asymmetric: all specialists share the exact same binary classification goal (Real vs. Fake), where the "Real" cues are relatively invariant across domains, but "Fake" artifacts are volatile and generator-dependent.
Methods
$R^2M$ leverages this structural asymmetry through a closed-form, training-free merging pipeline:
Specialists (Task Vectors τ_i = θ_i - θ_0)
│
┌───────────┴───────────┐
▼ ▼
Centered SVD Task Residuals
│ │
▼ ▼
Real Core (τ_core) Layerwise Rank Truncation (SVD-r)
│ │
│ Norm Matching across Tasks
│ │
└───────────┬───────────┘
▼
Merged Backbone:
θ* = θ_0 + τ_core + η · τ_merge^res
- Isolating the Real Core: Specialists are represented by their task vectors $\tau_i = \theta_i - \theta_0$ relative to a shared pretrained backbone $\theta_0$ (CLIP-L/14). By centering the task matrix and applying SVD, the dominant right singular vectors isolate the shared direction corresponding to the invariant "Real" feature subspace ($\tau_{\text{core}}$).
- Extracting and Denoising Fake Residuals: The mean-centered residuals $\delta_i = \tau_i - \bar{\tau}$ isolate generator-specific artifacts. To filter out nuisance interference and noise, layerwise SVD is applied to weight slices (in attention and MLP blocks) to retain only the top-$r$ components.
- Per-Task Norm Matching: Truncated residuals are rescaled to equal mean Euclidean norms before uniform averaging. This prevents any single specialist (e.g., one trained on a starkly obvious artifact) from dominating the parameter update.
- Unified Linear Head: The authors prove theoretically (under local linearity and bounded off-axis drift) that because the Real core pushes features along a common separation axis, the specialist classification heads are nearly collinear. Thus, a simple arithmetic average of specialist linear heads suffices without losing discriminative power.
Datasets
The authors benchmarked their framework on DF40, a standardized benchmark featuring 40 manipulation and generation methods across four categories:
- Face Swapping (FS) (e.g., FaceSwap, SimSwap, FSGAN)
- Face Reenactment (FR) (e.g., SadTalker, FOMM, HyperReenact)
- Entire Face Synthesis (EFS) (e.g., StyleGAN2/3/XL, DiT, PixArt, DDPM)
- Face Editing (FE)
Evaluations followed DF40's standardized protocols across FaceForensics++ (FF++) and Celeb-DF (CDF):
- Protocol 1 (Seen Domain & Categories): In-distribution evaluation on FF++.
- Protocol 2 (Cross-Domain, Seen Categories): Specialists trained on FF++ evaluated on Celeb-DF.
- Protocol 3 (Unseen Forgery & Domain): Zero-shot transfer to eight unseen engines (Midjourney, WhichIsReal, HeyGen, DeepFaceLab, StarGAN, StarGAN v2, StyleCLIP, CollabDiff).
- Heterogeneous Real Sets: Additional tests combining both FF++ and CDF real distributions.
Key Insights and Ideas
- Why Weight Averaging Naturally Favors "Real": Feature similarity probes revealed that standard weight averaging (WA) preserves representations on Real images while drastically attenuating generator-specific Fake cues (cosine similarity drops up to 25.4%). $R^2M$ builds directly on this intuition by explicitly preserving the Real core while preventing the destructive cancellation of Fake artifacts.
- Avoiding All-in-One Failure Modes: Jointly trained models frequently overfit to specific artifact types or catastrophically fail on completely new modalities (e.g., an all-in-one model collapsed to an AUC of 0.177 on Midjourney). Parameter-space merging maintains much higher worst-case floors.
- Zero-Retraining Composability: When a newly emerged generator appears (e.g., Midjourney or Uniface), operators can simply fine-tune one isolated specialist on that single generator and fold its residual into the deployed model in seconds, without touching the legacy data or modifying old specialist weights.
Key Results
- Seen Retention (Protocol 1 - FF++): $R^2M$ tied state-of-the-art CART with a mean AUC of 0.988, outperforming joint All-in-one training (0.979).
- Cross-Domain Transfer (Protocol 2 - CDF): When tested across domains, $R^2M$ achieved 0.919 mean AUC, substantially higher than CART (0.877), TIES (0.869), Weight Averaging (0.834), and All-in-one (0.662).
- Unseen Generalization (Protocol 3): On open-world unseen generators, $R^2M$ attained the highest macro-mean AUC (0.774 vs. All-in-one's 0.665 and best single specialist's 0.749), preventing severe outlier drops on diffusion generators like Midjourney (0.551 vs. 0.177).
- Multi-Source Real Distribution: Merging six specialists trained across mixed real sources (FF++ and CDF), $R^2M$ preserved higher accuracy on hard domains than competing mergers (0.901 vs. 0.867 on FS-ff).
Limitations and Future Work
- Security Against Untrusted Specialists: In open or decentralized pipelines, malicious actors could provide poisoned specialist weights designed to degrade the Real core or bypass detection for specific generators.
- Scale and Modality Bounds: The study evaluated models using a CLIP-L/14 image backbone; evaluating how $R^2M$ behaves across diverse vision backbones, multi-modal architectures (audio-visual deepfakes), or much larger specialist collections ($N \gg 10$) remains open.
- Hierarchical Merging: Future extensions could investigate self-expanding, hierarchical merging schemes that recursively cluster related generators into sub-families before composing top-level cores.
Summary updated