Introduction
The rapid proliferation of deepfake technology presents a significant challenge for digital forensics. As generative models evolve from simple face swaps to hyper-realistic, commercially available AI, the diversity of forgery artifacts grows exponentially. This "Model Explosion" (Figure 1) makes traditional joint training—where a single model is trained on all available datasets—increasingly impractical due to computational costs and the constant emergence of new generators. Furthermore, training on disparate forgery types often leads to task interference, where learning one set of artifacts degrades performance on another.
Methodology: Real-Aware Residual Model Merging ($R^2M$)
To address these challenges, we present Real-aware Residual Model Merging ($R^2M$), a training-free framework designed to combine multiple deepfake detection "specialists" into a single, robust "super-detector." The framework is predicated on the observation that while "Fake" cues are generator-specific and volatile, the representation of "Real" faces remains stable across datasets.
The $R^2M$ pipeline consists of four primary steps:
- Isolation of the 'Real' Core: Singular Value Decomposition (SVD) is applied to task vectors (the difference between specialist and base model weights) to identify shared directions representing "Real" features.
- Residual Extraction: Each specialist model is decomposed into this shared Real-aligned component and a "Fake" residual containing generator-specific artifacts.
- Denoising: Layerwise rank truncation is employed to clean the residuals, preserving informative signals while discarding noise.
- Norm-Matched Merging: Residuals are normalized and rescaled before being averaged, preventing any single specialist from dominating the final merged model.
This modular approach allows for "plug-and-play" integration: when a new forgery method emerges, a new specialist can be fine-tuned and its residual merged into the existing model without retraining the entire system.
Experimental Setup
The $R^2M$ framework was evaluated using the DF40 benchmark, encompassing 40 manipulation methods across categories such as Face Swapping, Face Reenactment, Entire Face Synthesis, and Face Editing. Performance was assessed across three protocols:
- Protocol 1 (In-distribution): Testing on the same domain as training (FF++).
- Protocol 2 (Cross-domain): Testing on unseen domains (e.g., Celeb-DF).
- Protocol 3 (Unseen forgeries): Testing on entirely new datasets like Midjourney and StyleCLIP to simulate real-world open-set conditions.
Results and Discussion
$R^2M$ demonstrates superior generalization capabilities. On unseen datasets (Protocol 3), the framework achieved a macro mean AUC of 0.774, significantly outperforming joint training (0.665) and other merging techniques such as TIES or CART.
A key advantage of $R^2M$ is its ability to maintain the performance of original specialists while gaining accuracy on new tasks. This plasticity allows the model to integrate detection capabilities for hyper-realistic faces generated by modern models like Sora and Stable Diffusion (Figure 2) without "forgetting" legacy forgery types.
Furthermore, the framework's scalability is evident as performance remains stable even as the number of merged specialists increases. Theoretically, the merging strategy aligns "Real-vs-Fake" separation vectors in feature space, allowing a simple linear classification head to effectively distinguish real from forged content across diverse generative sources.
Conclusion
Real-aware Residual Model Merging ($R^2M$) offers a scalable, efficient, and training-free solution for the evolving deepfake landscape. By isolating shared "Real" representations from idiosyncratic "Fake" artifacts, it achieves state-of-the-art robustness. Future research will focus on extending this framework to video-level temporal consistency and exploring multi-modal merging across different model architectures and data types.
Summary updated