Recap: Detecting Deepfake Video with Unpredictable Tampered Traces via Recovering Faces and Mapping Recovered Faces

Juan Hu, Xin Liao, Difei Gao, Satoshi Tsutsui, Qian Wang, Zheng Qin, Mike Zheng ShouarXiv 2023 · DOI · arXiv

Abstract

Deepfake generation methods are evolving rapidly, leaving behind unpredictable and random tampering traces that challenge standard detection models. This paper introduces Recap, a two-stage detection framework designed to enhance cross-domain generalization. Recap first reconstructs masked facial regions using a consistency-learning approach to expose inherent facial inconsistencies. It then maps these reconstructions to mathematically amplify the discrepancies between real and fake faces. By focusing on learning the structural consistency of real faces rather than specific artifacts, Recap achieves state-of-the-art performance across multiple benchmark datasets, including Celeb-DF and DFDC.

Introduction

As deepfake technology advances, the artifacts left behind by manipulation algorithms are becoming increasingly subtle and randomized. Traditional detection methods often function as specialized detectors, searching for specific, known forgery indicators such as unnatural blending boundaries or irregular blinking patterns.

However, modern deepfakes (e.g., Face2Face or FSGAN) alter random facial regions, leaving unpredictable traces. When a detector trained on one type of deepfake encounters a new, unseen manipulation method, it often fails—a phenomenon known as the cross-domain generalization problem. Recap addresses this by shifting the focus from "learning what is fake" to "learning what is real," identifying deepfakes by their lack of natural facial consistency.

Conceptual overview of the Recap framework, showing how invisible differences are amplified into obvious discrepancies through recovering and mapping stages.

Methodology

The Recap architecture is divided into two primary stages: Recovering for Consistency Learning and Mapping for Discrepancy Learning.

1. Recovering for Consistency Learning

Recap utilizes a Masked Autoencoder (MAE) based on Vision Transformers. Unlike standard MAE approaches that use random masking, Recap introduces an anatomy-aware masking strategy. Using facial landmarks, the model specifically masks Regions of Interest (ROIs) such as the eyes, nose, cheeks, and lips.

The autoencoder is trained exclusively on real faces. By masking 75% of these ROIs and requiring the model to reconstruct them, the network learns the inherent global consistency of human facial structures. When a fake face is processed, the decoder struggles to reconstruct it accurately because the manipulated regions lack this ground-truth consistency.

2. Mapping for Discrepancy Learning

The reconstructed faces are then passed through a ResNet-18 based mapping network. This stage acts as an amplifier, mapping well-reconstructed real faces to clean representations while forcing poorly reconstructed fake faces into even more distinct, "none-like" representations. This process artificially widens the mathematical gap between real and fake samples. To further improve robustness against unknown domains, the authors employ meta-learning during this stage, simulating cross-domain scenarios during training.

The full Recap pipeline, illustrating the two-stage process: (top) Recovering for Consistency Learning and (bottom) Mapping for Discrepancy Learning.

Experimental Results

The authors evaluated Recap using four benchmark datasets: FaceForensics++ (FF++), Celeb-DF, WildDeepfake (WildDF), and the Deepfake Detection Challenge (DFDC).

  • Cross-Domain Generalization: When trained on FF++ and tested on unseen datasets (Celeb-DF, WildDF, DFDC), Recap achieved an average AUC of 85.5%, significantly outperforming other generalized methods like OST (79.4%) and MRL (77.1%).
  • Intra-Domain Performance: On the FF++ dataset itself, Recap maintained a highly competitive average AUC of 99.0%.
  • Ablation Studies: Research into masking ratios revealed that a 75% ratio is optimal. Lower ratios (55%) fail to force the model to learn robust features, while higher ratios (95%) make the reconstruction task too difficult even for real faces.

Conclusion and Future Work

Recap demonstrates that focusing on the structural consistency of real faces is a powerful strategy for detecting unpredictable deepfakes. By amplifying reconstruction errors through a mapping network, the model achieves superior generalization across diverse datasets.

Current limitations include a reliance on precise landmark detection (using dlib), which may be vulnerable to extreme head poses or heavy occlusions. Future research may explore making the pipeline more robust to landmark failures and extending the framework to better capture temporal consistencies across longer video sequences.

Summary updated

Abstract

The exploitation of Deepfake techniques for malicious intentions has driven significant research interest in Deepfake detection. Deepfake manipulations frequently introduce random tampered traces, leading to unpredictable outcomes in different facial regions. However, existing detection methods heavily rely on specific forgery indicators, and as the forgery mode improves, these traces become increasingly randomized, resulting in a decline in the detection performance of methods reliant on specific forgery traces. To address the limitation, we propose Recap, a novel Deepfake detection model that exposes unspecific facial part inconsistencies by recovering faces and enlarges the differences between real and fake by mapping recovered faces. In the recovering stage, the model focuses on randomly masking regions of interest (ROIs) and reconstructing real faces without unpredictable tampered traces, resulting in a relatively good recovery effect for real faces while a poor recovery effect for fake faces. In the mapping stage, the output of the recovery phase serves as supervision to guide the facial mapping process. This mapping process strategically emphasizes the mapping of fake faces with poor recovery, leading to a further deterioration in their representation, while enhancing and refining the mapping of real faces with good representation. As a result, this approach significantly amplifies the discrepancies between real and fake videos. Our extensive experiments on standard benchmarks demonstrate that Recap is effective in multiple scenarios.
BibTeX
@misc{hu2023recap,
  author = {Juan Hu and Xin Liao and Difei Gao and Satoshi Tsutsui and Qian Wang and Zheng Qin and Mike Zheng Shou},
  title = {{Recap: Detecting Deepfake Video with Unpredictable Tampered Traces via Recovering Faces and Mapping Recovered Faces}},
  year = {2023},
  eprint = {2308.09921},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.09921},
}
Figure