DFREC: DeepFake Identity Recovery Based on Identity-aware Masked Autoencoder

Peipeng Yu, Hui Gao, Zhitao Huang, Zhihua Xia, Chip-Hong Chang

Abstract

Recent advances in deepfake forensics have primarily focused on improving classification accuracy and generalization. However, existing algorithms often lack intuitive interpretability and identity traceability, which are crucial for forensic investigations. We introduce the DeepFake Identity Recovery scheme (DFREC), a novel framework designed to recover both source and target faces from a single deepfake image. DFREC utilizes an Identity Segmentation Module (ISM), a Source Identity Reconstruction Module (SIRM), and a Target Identity Reconstruction Module (TIRM) featuring an identity-aware Masked Autoencoder. Our evaluations across six high-fidelity face-swapping attacks demonstrate that DFREC achieves superior recovery performance and high fidelity compared to state-of-the-art methods.

Introduction

The rise of sophisticated deepfake technology has created a pressing need for advanced forensic tools. While current methods excel at detecting whether an image is manipulated, they rarely provide insights into the identities involved. For forensic investigators, identifying the "source" (the person providing facial features) and the "target" (the original person in the video) is essential for establishing a chain of evidence. DFREC addresses this by shifting the focus from binary detection to comprehensive identity tracing, allowing for the restoration of original identities involved in a face-swap.

Methodology

The DFREC framework operates through three specialized modules that work in sequence to decouple and reconstruct identities.

Identity Segmentation Module (ISM)

The ISM employs a DeepLabv3 architecture to generate a probability map of the forged image. This map distinguishes between pixels belonging to the "source" identity (the swap) and the "target" background context, providing a spatial guide for subsequent reconstruction.

Source Identity Reconstruction Module (SIRM)

Using the segmented source information, the SIRM reconstructs the source face. A key insight of this module is that the forged region often retains "latent target identity features"—subtle cues of the original target's identity that persist despite the swap due to the inherent correlation between identity and attributes like pose and expression. The SIRM extracts these features to aid in target reconstruction.

Target Identity Reconstruction Module (TIRM)

The TIRM is the core of the recovery process, utilizing an Identity-aware Masked Autoencoder (MAE):

  • Semantic-Guided Masking: Unlike standard MAEs that mask random patches, DFREC uses the ISM's probability map to specifically mask out the forged source patches.
  • Identity Fusion: A Vision Transformer (ViT) encoder processes the unmasked background, while an Identity Fusion Decoder merges these features with the latent target features from the SIRM to "inpaint" the original target face with high fidelity.

The model is optimized using a quadruple loss function encompassing Identity Loss (FaceNet), Perceptual Loss (VGG), Attribute Loss, and Patch Recovery Loss.

Experimental Evaluation

DFREC was evaluated on three major benchmarks: FaceForensics++, CelebaMegaFS, and FFHQ-E4S, covering various forgery algorithms such as DeepFake (DF), FaceShifter (FShi), and IDInjection (IDI).

Performance Analysis

DFREC significantly outperforms state-of-the-art methods like DFI and Delocate. On the CelebaMegaFS (FTM) dataset, DFREC achieved an Identity Similarity (IDSim) score of 0.7192, nearly doubling the performance of DFI (0.4137).

Identity similarity distribution for the CelebaMegaFS-FTM dataset, showing the model's ability to distinguish target and source identities from unrelated ones.

In terms of image quality, DFREC achieved a Frechet Inception Distance (FID) of 8.34 on the FFHQ-E4S dataset, outperforming the next best method, MAT (9.82). The framework also demonstrates robust generalization, maintaining high performance even on unseen forgery types. In intra-dataset testing, DFREC achieved near-perfect identity recovery accuracy ($Acc_{id}$) ranging from 98% to 100%.

Identity similarity distribution for the FaceForensics++ DeepFake algorithm, illustrating consistent performance across different benchmarks.

Conclusion

DFREC proves that treating target face recovery as an informed inpainting task effectively avoids "identity leakage"—a common issue where source features persist in the recovered target face. Furthermore, the ability to recover pristine faces provides an interpretable signal for forgery detection: if the recovered identities do not align with the input, the image is likely a deepfake. Future research will focus on enhancing robustness to "in-the-wild" conditions, such as extreme poses or poor lighting, and integrating these capabilities into real-time forensic pipelines.

Summary updated

Abstract

Recent advances in deepfake forensics have primarily focused on improving the classification accuracy and generalization performance. Despite enormous progress in detection accuracy across a wide variety of forgery algorithms, existing algorithms lack intuitive interpretability and identity traceability to help with forensic investigation. In this paper, we introduce a novel DeepFake Identity Recovery scheme (DFREC) to fill this gap. DFREC aims to recover the pair of source and target faces from a deepfake image to facilitate deepfake identity tracing and reduce the risk of deepfake attack. It comprises three key components: an Identity Segmentation Module (ISM), a Source Identity Reconstruction Module (SIRM), and a Target Identity Reconstruction Module (TIRM). The ISM segments the input face into distinct source and target face information, and the SIRM reconstructs the source face and extracts latent target identity features with the segmented source information. The background context and latent target identity features are synergetically fused by a Masked Autoencoder in the TIRM to reconstruct the target face. We evaluate DFREC on six different high-fidelity face-swapping attacks on FaceForensics++, CelebaMegaFS and FFHQ-E4S datasets, which demonstrate its superior recovery performance over state-of-the-art deepfake recovery algorithms. In addition, DFREC is the only scheme that can recover both pristine source and target faces directly from the forgery image with high fadelity.
BibTeX
@misc{yu2024dfrec,
  author = {Peipeng Yu and Hui Gao and Zhitao Huang and Zhihua Xia and Chip-Hong Chang},
  title = {{DFREC: DeepFake Identity Recovery Based on Identity-aware Masked Autoencoder}},
  year = {2024},
  eprint = {2412.07260},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.07260},
}
Figure