Real-Aware Residual Model Merging for Deepfake Detection

Jinhee Park, Guisik Kim, Choongsang Cho, Junseok KwonSubmission to ICLR 2026 · Rejected

TL;DR

Retraining deepfake detectors every time a new generative model emerges is computationally prohibitive, while multi-task joint training often suffers from negative task interference. Real-aware Residual Model Merging ($R^2M$) introduces a training-free parameter-space merging technique that disentangles a shared "Real" representation from generator-specific "Fake" residuals using low-rank factorization. By retaining a single core Real component and aggregating denoised, norm-matched Fake residuals, $R^2M$ matches or beats joint training on seen forgeries and significantly improves zero-shot transfer to unseen deepfake generators.

What Problem Does This Paper Address?

Generative video and image synthesis (diffusion models, talking heads, advanced face-swapping) evolves at a breakneck pace. Defending against deepfakes usually faces an operational dilemma:

  1. Full Retraining / Joint Training: Gathering data from every new generator and retraining an "all-in-one" detector is slow, costly, and vulnerable to catastrophic forgetting or cross-task interference.
  2. Specialist Proliferation: Training independent specialist detectors for each generator family creates an unmanageable ensemble footprint in production.

Model merging—combining fine-tuned weights directly in parameter space without gradient descent—offers an appealing middle ground. However, standard merging algorithms (e.g., simple weight averaging, Task Arithmetic, TIES) were built for multi-task problems with disjoint classification labels. In contrast, deepfake detection is inherently asymmetric: all specialists share the exact same binary classification goal (Real vs. Fake), where the "Real" cues are relatively invariant across domains, but "Fake" artifacts are volatile and generator-dependent.

Methods

$R^2M$ leverages this structural asymmetry through a closed-form, training-free merging pipeline:

Specialists (Task Vectors τ_i = θ_i - θ_0)
                     │
         ┌───────────┴───────────┐
         ▼                       ▼
  Centered SVD             Task Residuals
         │                       │
         ▼                       ▼
   Real Core (τ_core)      Layerwise Rank Truncation (SVD-r)
         │                       │
         │                 Norm Matching across Tasks
         │                       │
         └───────────┬───────────┘
                     ▼
           Merged Backbone:
       θ* = θ_0 + τ_core + η · τ_merge^res
  • Isolating the Real Core: Specialists are represented by their task vectors $\tau_i = \theta_i - \theta_0$ relative to a shared pretrained backbone $\theta_0$ (CLIP-L/14). By centering the task matrix and applying SVD, the dominant right singular vectors isolate the shared direction corresponding to the invariant "Real" feature subspace ($\tau_{\text{core}}$).
  • Extracting and Denoising Fake Residuals: The mean-centered residuals $\delta_i = \tau_i - \bar{\tau}$ isolate generator-specific artifacts. To filter out nuisance interference and noise, layerwise SVD is applied to weight slices (in attention and MLP blocks) to retain only the top-$r$ components.
  • Per-Task Norm Matching: Truncated residuals are rescaled to equal mean Euclidean norms before uniform averaging. This prevents any single specialist (e.g., one trained on a starkly obvious artifact) from dominating the parameter update.
  • Unified Linear Head: The authors prove theoretically (under local linearity and bounded off-axis drift) that because the Real core pushes features along a common separation axis, the specialist classification heads are nearly collinear. Thus, a simple arithmetic average of specialist linear heads suffices without losing discriminative power.

Datasets

The authors benchmarked their framework on DF40, a standardized benchmark featuring 40 manipulation and generation methods across four categories:

  • Face Swapping (FS) (e.g., FaceSwap, SimSwap, FSGAN)
  • Face Reenactment (FR) (e.g., SadTalker, FOMM, HyperReenact)
  • Entire Face Synthesis (EFS) (e.g., StyleGAN2/3/XL, DiT, PixArt, DDPM)
  • Face Editing (FE)

Evaluations followed DF40's standardized protocols across FaceForensics++ (FF++) and Celeb-DF (CDF):

  • Protocol 1 (Seen Domain & Categories): In-distribution evaluation on FF++.
  • Protocol 2 (Cross-Domain, Seen Categories): Specialists trained on FF++ evaluated on Celeb-DF.
  • Protocol 3 (Unseen Forgery & Domain): Zero-shot transfer to eight unseen engines (Midjourney, WhichIsReal, HeyGen, DeepFaceLab, StarGAN, StarGAN v2, StyleCLIP, CollabDiff).
  • Heterogeneous Real Sets: Additional tests combining both FF++ and CDF real distributions.

Key Insights and Ideas

  • Why Weight Averaging Naturally Favors "Real": Feature similarity probes revealed that standard weight averaging (WA) preserves representations on Real images while drastically attenuating generator-specific Fake cues (cosine similarity drops up to 25.4%). $R^2M$ builds directly on this intuition by explicitly preserving the Real core while preventing the destructive cancellation of Fake artifacts.
  • Avoiding All-in-One Failure Modes: Jointly trained models frequently overfit to specific artifact types or catastrophically fail on completely new modalities (e.g., an all-in-one model collapsed to an AUC of 0.177 on Midjourney). Parameter-space merging maintains much higher worst-case floors.
  • Zero-Retraining Composability: When a newly emerged generator appears (e.g., Midjourney or Uniface), operators can simply fine-tune one isolated specialist on that single generator and fold its residual into the deployed model in seconds, without touching the legacy data or modifying old specialist weights.

Key Results

  • Seen Retention (Protocol 1 - FF++): $R^2M$ tied state-of-the-art CART with a mean AUC of 0.988, outperforming joint All-in-one training (0.979).
  • Cross-Domain Transfer (Protocol 2 - CDF): When tested across domains, $R^2M$ achieved 0.919 mean AUC, substantially higher than CART (0.877), TIES (0.869), Weight Averaging (0.834), and All-in-one (0.662).
  • Unseen Generalization (Protocol 3): On open-world unseen generators, $R^2M$ attained the highest macro-mean AUC (0.774 vs. All-in-one's 0.665 and best single specialist's 0.749), preventing severe outlier drops on diffusion generators like Midjourney (0.551 vs. 0.177).
  • Multi-Source Real Distribution: Merging six specialists trained across mixed real sources (FF++ and CDF), $R^2M$ preserved higher accuracy on hard domains than competing mergers (0.901 vs. 0.867 on FS-ff).

Limitations and Future Work

  • Security Against Untrusted Specialists: In open or decentralized pipelines, malicious actors could provide poisoned specialist weights designed to degrade the Real core or bypass detection for specific generators.
  • Scale and Modality Bounds: The study evaluated models using a CLIP-L/14 image backbone; evaluating how $R^2M$ behaves across diverse vision backbones, multi-modal architectures (audio-visual deepfakes), or much larger specialist collections ($N \gg 10$) remains open.
  • Hierarchical Merging: Future extensions could investigate self-expanding, hierarchical merging schemes that recursively cluster related generators into sub-families before composing top-level cores.

Summary updated

Abstract

Deepfake generators evolve quickly, making exhaustive data collection and repeated retraining impractical. We argue that model merging is a natural fit for deepfake detection: unlike generic multi-task settings with disjoint labels, deepfake specialists share the same binary decision and differ in generator-specific artifacts. Empirically, we show that simple weight averaging preserves Real representations while attenuating Fake-specific cues. Building upon these findings, we propose Real-aware Residual Model Merging (R$^2$M), a training-free parameter-space merging framework. R$^2$M estimates a shared Real component via a low-rank factorization of task vectors, decomposes each specialist into a Real-aligned part and a Fake residual, denoises residuals with layerwise rank truncation, and aggregates them with per-task norm matching to prevent any single generator from dominating. A concise rationale explains why a simple head suffices: the Real component induces a common separation direction in feature space, while truncated residuals contribute only minor off-axis variations. Across in-distribution, cross-dataset, and unseen-dataset, R$^2$M outperforms joint training and other merging baselines. Importantly, R$^2$M is also composable: when a new forgery family appears, we fine-tune one specialist and re-merge, eliminating the need for retraining.
BibTeX
@misc{park2025real,
  author = {Jinhee Park and Guisik Kim and Choongsang Cho and Junseok Kwon},
  title = {{Real-Aware Residual Model Merging for Deepfake Detection}},
  year = {2025},
  eprint = {2509.24367},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.24367},
}
Figure