Real-Aware Residual Model Merging for Deepfake Detection

Jinhee Park, Guisik Kim, Choongsang Cho, Junseok Kwon

Introduction

The rapid proliferation of deepfake technology presents a significant challenge for digital forensics. As generative models evolve from simple face swaps to hyper-realistic, commercially available AI, the diversity of forgery artifacts grows exponentially. This "Model Explosion" (Figure 1) makes traditional joint training—where a single model is trained on all available datasets—increasingly impractical due to computational costs and the constant emergence of new generators. Furthermore, training on disparate forgery types often leads to task interference, where learning one set of artifacts degrades performance on another.

The rapid evolution and 'Model Explosion' of generative deepfake technologies.

Methodology: Real-Aware Residual Model Merging ($R^2M$)

To address these challenges, we present Real-aware Residual Model Merging ($R^2M$), a training-free framework designed to combine multiple deepfake detection "specialists" into a single, robust "super-detector." The framework is predicated on the observation that while "Fake" cues are generator-specific and volatile, the representation of "Real" faces remains stable across datasets.

The $R^2M$ pipeline consists of four primary steps:

  1. Isolation of the 'Real' Core: Singular Value Decomposition (SVD) is applied to task vectors (the difference between specialist and base model weights) to identify shared directions representing "Real" features.
  2. Residual Extraction: Each specialist model is decomposed into this shared Real-aligned component and a "Fake" residual containing generator-specific artifacts.
  3. Denoising: Layerwise rank truncation is employed to clean the residuals, preserving informative signals while discarding noise.
  4. Norm-Matched Merging: Residuals are normalized and rescaled before being averaged, preventing any single specialist from dominating the final merged model.

This modular approach allows for "plug-and-play" integration: when a new forgery method emerges, a new specialist can be fine-tuned and its residual merged into the existing model without retraining the entire system.

Experimental Setup

The $R^2M$ framework was evaluated using the DF40 benchmark, encompassing 40 manipulation methods across categories such as Face Swapping, Face Reenactment, Entire Face Synthesis, and Face Editing. Performance was assessed across three protocols:

  • Protocol 1 (In-distribution): Testing on the same domain as training (FF++).
  • Protocol 2 (Cross-domain): Testing on unseen domains (e.g., Celeb-DF).
  • Protocol 3 (Unseen forgeries): Testing on entirely new datasets like Midjourney and StyleCLIP to simulate real-world open-set conditions.

Results and Discussion

$R^2M$ demonstrates superior generalization capabilities. On unseen datasets (Protocol 3), the framework achieved a macro mean AUC of 0.774, significantly outperforming joint training (0.665) and other merging techniques such as TIES or CART.

A key advantage of $R^2M$ is its ability to maintain the performance of original specialists while gaining accuracy on new tasks. This plasticity allows the model to integrate detection capabilities for hyper-realistic faces generated by modern models like Sora and Stable Diffusion (Figure 2) without "forgetting" legacy forgery types.

Hyper-realistic faces generated by modern models like Sora and Stable Diffusion.

Furthermore, the framework's scalability is evident as performance remains stable even as the number of merged specialists increases. Theoretically, the merging strategy aligns "Real-vs-Fake" separation vectors in feature space, allowing a simple linear classification head to effectively distinguish real from forged content across diverse generative sources.

Conclusion

Real-aware Residual Model Merging ($R^2M$) offers a scalable, efficient, and training-free solution for the evolving deepfake landscape. By isolating shared "Real" representations from idiosyncratic "Fake" artifacts, it achieves state-of-the-art robustness. Future research will focus on extending this framework to video-level temporal consistency and exploring multi-modal merging across different model architectures and data types.

Summary updated

Abstract

Deepfake generators evolve quickly, making exhaustive data collection and repeated retraining impractical. We argue that model merging is a natural fit for deepfake detection: unlike generic multi-task settings with disjoint labels, deepfake specialists share the same binary decision and differ in generator-specific artifacts. Empirically, we show that simple weight averaging preserves Real representations while attenuating Fake-specific cues. Building upon these findings, we propose Real-aware Residual Model Merging (R$^2$M), a training-free parameter-space merging framework. R$^2$M estimates a shared Real component via a low-rank factorization of task vectors, decomposes each specialist into a Real-aligned part and a Fake residual, denoises residuals with layerwise rank truncation, and aggregates them with per-task norm matching to prevent any single generator from dominating. A concise rationale explains why a simple head suffices: the Real component induces a common separation direction in feature space, while truncated residuals contribute only minor off-axis variations. Across in-distribution, cross-dataset, and unseen-dataset, R$^2$M outperforms joint training and other merging baselines. Importantly, R$^2$M is also composable: when a new forgery family appears, we fine-tune one specialist and re-merge, eliminating the need for retraining.
BibTeX
@misc{park2025real,
  author = {Jinhee Park and Guisik Kim and Choongsang Cho and Junseok Kwon},
  title = {{Real-Aware Residual Model Merging for Deepfake Detection}},
  year = {2025},
  eprint = {2509.24367},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.24367},
}
Figure