Robust Face-Swap Detection Based on 3D Facial Shape Information

Weinan Guan, Wei Wang, Jing Dong, Bo Peng, Tieniu Tan

Abstract

Deepfake detection often relies on pixel-level artifacts that are easily lost during video compression or "laundering." This paper proposes a robust detection framework based on 3D facial shape information. By using 3D Morphable Models (3DMM) to extract identity-specific geometric features, the system identifies inconsistencies between a person's observed appearance and their underlying 3D facial structure. This semantic approach proves highly resilient to image degradation and provides interpretable results, outperforming standard pixel-based models in challenging real-world conditions.

Introduction

Most current Deepfake detection methods are "pixel-level" detectors. They look for tiny mathematical inconsistencies, blending boundaries, or frequency artifacts left behind by AI generation. While effective on high-quality original files, these methods often fail in the real world. When a video is "laundered"—compressed for social media, resized, or hit with digital noise—those subtle pixel artifacts disappear, and the detectors fail. Furthermore, these models often lack interpretability; they tell you a video is fake but can't explain why in human-understandable terms.

This research addresses the need for a semantic clue that is robust to image degradation and provides a clear biological reason for a "fake" verdict: the mismatch between a person’s 3D facial structure and their surface appearance.

Methodology

The proposed framework is based on biometric inconsistency. Unlike standard methods that hunt for blurry pixels, this approach uses 3D Morphable Models (3DMM) to prove that while a face might look like a specific celebrity, the underlying "bone structure" belongs to someone else.

The proposed detection pipeline: 3D facial shape is extracted and compared against a reference template using Mahalanobis distance.

As shown in the pipeline above, the system follows a multi-step logic:

  1. 3D Shape Extraction: The system uses a 3DMM to estimate the 3D shape of a face from a single 2D image, separating the face into identity (permanent shape) and expression (temporary movement).
  2. The Template: For a "key figure" (like a world leader), the system creates a reference template of their actual 3D facial shape using verified footage.
  3. The Inconsistency Clue: In a face-swap, the AI replaces the appearance (skin texture, eyes, nose details) of the target with the source's features, but the underlying 3D geometry often remains that of the original target.
  4. Mahalanobis Distance: To decide if a face is a match, the system calculates the Mahalanobis Distance between the 3D shape in the video and the person's registered template. This metric accounts for the natural distribution and variance of the person's facial features, making it much more accurate at distinguishing between individuals than simple Euclidean distance.

Experimental Evaluation

Datasets and Robustness Testing

The study utilized several benchmarks, including FaceForensics++, which contains 1,000 pristine videos and their corresponding fakes. To test real-world applicability, the authors manually applied "laundering attacks" such as heavy JPEG compression, Gaussian noise, and blurring.

Example of a 'laundered' image with heavy digital noise used to test the robustness of the detector.

Key Results

The proposed method significantly outperformed industry-standard models like XceptionNet and EfficientNet in challenging conditions:

  • Robustness to Noise: On data with heavy Gaussian noise, the proposed method maintained an accuracy of ~85.7%, while XceptionNet plummeted to ~79%.
  • Cross-Domain Performance: When trained on one type of fake (FaceSwap) and tested on another (Deepfakes), the proposed method achieved ~82.1% accuracy, whereas pixel-based models struggled to stay above ~47% (essentially random guessing).
  • Wild Testing: In a test case using Donald Trump footage, the model successfully separated genuine clips from fakes with a clear threshold, regardless of lighting or background.

Discussion and Conclusion

The core insight of this work is that face-swapping is essentially a digital "skin graft." While the skin looks like Person A, the skull structure (represented by 3DMM parameters) remains Person B. Because 3D shape is a high-level semantic feature, it survives heavy video compression and noise that would normally "wash away" the evidence used by pixel-based detectors.

Limitations and Future Work

This is not a "blind" detector; it requires a pre-built 3D shape template for the individual being protected. It is designed for high-profile figures rather than general-purpose detection. Additionally, performance is capped by the accuracy of the 3D reconstruction. Future directions include moving toward more advanced facial shape estimation methods to improve the resolution of the biometric check, potentially allowing for even more subtle detection.

Summary updated

BibTeX
@inproceedings{guan2022robust,
  author = {Weinan Guan and Wei Wang and Jing Dong and Bo Peng and Tieniu Tan},
  title = {{Robust Face-Swap Detection Based on 3D Facial Shape Information}},
  booktitle = {CICAI},
  year = {2022},
  url = {https://openreview.net/forum?id=2CPVa3LUkB},
}
Figure