Delocate: Detection and Localization for Deepfake Videos with Randomly-Located Tampered Traces

Juan Hu, Xin Liao, Difei Gao, Satoshi Tsutsui, Qian Wang, Zheng Qin, Mike Zheng Shou

Abstract

Deepfake detection faces significant challenges in generalization and explainability as forgery techniques evolve to produce increasingly subtle and randomly distributed artifacts. This paper introduces Delocate, a novel two-stage framework designed to bridge the generalization gap by focusing on the intrinsic properties of real faces rather than the specific artifacts of known forgeries. By combining a facial-part-aware reconstruction task with a meta-learning-based localization stage, Delocate achieves state-of-the-art performance in both identifying and precisely pinpointing tampered regions across unseen domains.

Introduction

As Deepfake technology advances, the resulting "forgery traces" have become increasingly difficult to detect. Most existing detection methods suffer from two primary limitations. First, they exhibit a significant generalization gap; models trained on specific datasets (e.g., FaceForensics++) often fail when tested on different datasets (e.g., Celeb-DF) because they overfit to specific forgery patterns. Second, many detectors lack explainability, providing a binary classification score without indicating where or why a video was flagged as manipulated.

Delocate addresses these issues by shifting the focus from "learning fakes" to "learning reality." By understanding the structural and temporal consistency of authentic human faces, the model can identify anomalies in manipulated videos regardless of the specific generation technique used.

Methodology

The Delocate framework operates in two distinct stages: Consistency Learning through reconstruction and Discrepancy Learning through localization.

Stage 1: Recovering for Consistency Learning

In the first stage, the model learns the underlying distribution of authentic facial data. Unlike traditional detectors that use both real and fake samples, Delocate utilizes a Masked Autoencoder (MAE) trained exclusively on real faces.

Instead of the standard random patch masking used in general-purpose MAEs, the authors implement a Facial-Part Masking Strategy. This strategy targets specific Regions of Interest (ROIs) such as the eyes, nose, and lips. By forcing the model to reconstruct these specific components, it learns the complex structural and temporal relationships inherent in human facial harmony. The logic is that a model trained to be an expert in reconstructing real faces will naturally produce high reconstruction errors when presented with tampered areas, as those areas violate the learned consistency of real data.

Stage 2: Localization for Discrepancy Learning

The second stage fine-tunes the model to amplify the differences between real and fake samples.

  • Mapping Module: This component processes the reconstruction results from Stage 1, mapping them into a feature space where the discrepancies in fake faces are highlighted.
  • Meta-Learning Strategy: To prevent the model from memorizing specific training fakes, the authors employ a meta-learning approach. By splitting data into meta-train and meta-test sets, the model simulates encounters with "unseen" domains during training, encouraging the development of more generalized features.
  • Encoder-Decoder Localization: A dedicated branch predicts a pixel-level mask of the tampered area. This localization task serves as a guide for the final classification, ensuring that the model's decision is grounded in specific, detectable anomalies.

Experimental Results

The researchers evaluated Delocate against four major benchmarks: FaceForensics++ (FF++), Celeb-DF (CDF), DeeperForensics-1.0 (DFo), and the Deepfake Detection Challenge (DFDC).

Cross-Dataset Generalization

Delocate demonstrated superior robustness in cross-domain scenarios. When trained on FF++ and tested on the high-quality Celeb-DF dataset, Delocate achieved an AUC of 91.3%, a substantial improvement over previous state-of-the-art methods which typically hovered around 86.2%.

Localization Precision

Using the Intersection over Union (IoU) metric, Delocate outperformed existing localization frameworks. The model produced cleaner and more accurate heatmaps of tampered regions, providing a clear visual explanation for its detection results. Furthermore, the model remained robust even when trained on a single type of manipulation (e.g., Face2Face) and tested on entirely different forgery methods.

Discussion and Insights

Several key insights emerged from the development of Delocate:

  1. Real-Only Pretraining as a Defense: Training the initial reconstruction stage only on real faces prevents the model from inheriting the biases of specific deepfake generators.
  2. Synergy Between Tasks: The authors found that the tasks of localization and detection are mutually beneficial; forcing the model to locate the forgery actually improves its overall classification accuracy.
  3. Domain-Specific Masking: Masking specific facial parts is more effective than random block masking, as it respects the biological and structural constraints of the human face.

Limitations and Future Work

While Delocate excels in generalization, it faces a minor "generalization vs. specialization" trade-off, showing a very slight performance decrease (0.2%) on intra-dataset tests compared to models specifically optimized for those datasets. Additionally, the two-stage architecture is computationally more intensive than simple binary classifiers. Future research will focus on refining temporal consistency to further improve performance against sophisticated, high-frame-rate video forgeries.

Conclusion

Delocate represents a significant step forward in making Deepfake detection more robust and explainable. By prioritizing the reconstruction of real facial features and utilizing meta-learning for localization, the framework effectively bridges the gap between laboratory training and real-world application.

Summary updated

BibTeX
@inproceedings{hu2024delocate,
  author = {Juan Hu and Xin Liao and Difei Gao and Satoshi Tsutsui and Qian Wang and Zheng Qin and Mike Zheng Shou},
  title = {{Delocate: Detection and Localization for Deepfake Videos with Randomly-Located Tampered Traces}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure