Abstract
The rampant use of forgery techniques poses a significant threat to the security of celebrities' identities. Although current deepfake detection methods have shown effectiveness when dealing with specific public face forgery datasets, their reliability diminishes when applied to open data. Moreover, these methods are susceptible to re-compression and mainly rely on pixel-level abnormalities in forgery faces. In this study, we present a novel approach to detecting face forgery by leveraging individual speaking patterns of facial expressions and head movements. Our method utilizes potential motion patterns and inter-frame variations to effectively differentiate between fake and real videos. We propose an end-to-end dual-branch detection network, named the spatial-temporal transformer (STT), which aims to safeguard the identity of the person-of-interest (POI) from deepfaking. The STT incorporates the spatial transformer (ST) to establish the connection between facial expressions and head movements, while the temporal transformer (TT) exploits inconsistencies in facial attribute changes. Additionally, we introduce a central compression loss to enhance the detection performance. Extensive experiments are conducted to evaluate the effectiveness of the STT, and the results demonstrate its superiority over other SOTA methods in detecting forgery videos involving POIs. Furthermore, our network exhibits resilience to pixel-level re-compression perturbations, making it a robust solution in the face of evolving forgery techniques.
Introduction: Beyond Pixel Artifacts
Most current deepfake detectors act as "artifact hunters," searching for microscopic pixel-level abnormalities or blending boundaries. While effective on high-quality benchmarks, these methods face two critical challenges: sensitivity to video compression (which often washes away pixel artifacts) and poor generalization to unseen forgery techniques.
This research proposes a shift in strategy: focusing on the idiosyncratic speaking patterns of a specific Person-of-Interest (POI). By analyzing the unique coordination between facial expressions, head movements, and eye gaze, the proposed Spatial-Temporal Transformer (STT) identifies forgeries that fail to replicate the natural behavioral "signature" of an individual.
Methodology: The STT Network
The STT is an end-to-end dual-branch network designed to capture high-level semantic inconsistencies rather than easily obscured pixel glitches.
Feature Representation
Instead of raw pixels, the model uses OpenFace to extract 31 high-level parameters per frame: 17 Facial Action Units (AUs), 6 head pose coordinates, and 8 parameters for 3D eye gaze. To capture subtle nuances, a motion amplification module is applied to the input video before feature extraction.
Dual-Branch Transformers
- Spatial Transformer (ST) Branch: This branch learns the "potential motion patterns" within a single frame, modeling how different facial features correlate (e.g., ensuring a specific mouth shape matches the correct cheek muscle movement).
- Temporal Transformer (TT) Branch: This branch focuses on "inter-frame variations" using masked multi-head attention to spot jerky or unnatural transitions over time that suggest forgery.
Central Compression Loss ($L_{cc}$)
To improve generalization and clear the decision boundary, the authors introduced a novel Central Compression Loss. This function pulls the feature representations of "real" videos closer together in the latent space while pushing "fake" videos further away.
Experimental Results
The study utilized the standard FaceForensics++ benchmark and a new DeepPOI dataset containing over 63,000 videos of world leaders faked using methods like SimSwap and Wav2lip.
- Superior Performance: On the DeepPOI dataset, STT achieved an accuracy of 96.87% and an AUC of 98.28%, outperforming the best existing POI-based methods by over 5%.
- Robustness to Compression: When tested on low-quality videos compressed with H.264, STT’s performance remained stable (dropping only ~3%), whereas artifact-based methods saw significant accuracy declines.
- Ablation Insights: The Spatial branch proved vital; its removal caused a 20% drop in accuracy, confirming that the relationship between different facial muscles is a key indicator of authenticity.
Conclusion
By moving the detection "battleground" from pixel-level artifacts to high-level semantic features like eye gaze and head pose, the STT network provides a robust defense against sophisticated deepfakes. While localized forgeries like lip-syncing remain a challenge, future integration of audio-visual phonetic patterns could further strengthen the system's resilience.
Summary updated