Abstract
Deepfake technology poses a significant challenge to digital integrity, necessitating robust detection mechanisms that can generalize across diverse manipulation techniques. This paper introduces a Video Transformer architecture that utilizes 3D-reconstructed UV texture maps alongside 2D facial frames to capture both spatial and temporal inconsistencies. Furthermore, an incremental learning strategy is implemented to allow the model to adapt to emerging deepfake methods with minimal additional training data. The proposed approach achieves state-of-the-art performance on major benchmarks, including an AUC of 99.79% on FaceForensics++.
Introduction
The rapid evolution of deepfake generation techniques has outpaced many current detection systems. Existing CNN-based detectors often suffer from three primary limitations: a failure to capture temporal inconsistencies (such as flickering), alignment issues when subjects are not facing the camera directly, and poor generalization to unseen datasets. To address these gaps, we propose a multi-modal transformer-based approach that "frontalizes" facial features using 3D UV texture maps and employs incremental learning to maintain performance across evolving threats.
Methodology
The proposed pipeline consists of several integrated stages designed to extract comprehensive feature sets:
3D UV Texture Reconstruction
To mitigate alignment issues, a 3D Dense Face Alignment (3DDFA) model is used to unwrap 2D facial frames into normalized UV texture maps. This process ensures that facial landmarks remain in consistent spatial coordinates regardless of head pose, providing a stable input for feature extraction.
Hybrid Feature Extraction and Transformer Integration
Features are extracted from both the original cropped faces and the UV texture maps using a pre-trained XceptionNet backbone. These features are then processed by a Transformer consisting of 12 encoder blocks. A key innovation is the use of Segment Embeddings—a technique borrowed from NLP models like BERT—which allow the model to distinguish between the 2D image data and the UV texture data within the same sequence.
Incremental Learning Strategy
To ensure the model remains effective against new manipulation types without "catastrophic forgetting," an incremental learning strategy is employed. A specialized loss function balances the classification of new data with the preservation of previously learned weights, enabling high performance on new datasets with significantly less training data than traditional methods.
Experimental Setup and Datasets
The model was evaluated on several industry-standard benchmarks, including FaceForensics++ (FF++), which contains sub-datasets like FaceSwap and Face2Face; the DeepFake Detection (DFD) dataset; and the large-scale Deepfake Detection Challenge (DFDC).
Results and Discussion
The Video Transformer demonstrates superior performance across all tested benchmarks. Notably, the hybrid model using segment embeddings achieved an AUC of 99.79% and an accuracy of 99.52% on the FaceForensics++ dataset.
On the challenging DFDC dataset, the model reached 91.69% accuracy, significantly outperforming standard detectors like MesoNet. Furthermore, the incremental learning approach proved highly efficient; for instance, the model achieved 99.28% accuracy on the Face2Face test using only 0.5% of the training data size required by competing architectures.
Conclusion
By combining 3D facial reconstruction with the temporal modeling capabilities of Transformers, this work provides a robust framework for deepfake detection. The inclusion of UV maps provides a "frontalized" view that stabilizes texture analysis, while the incremental learning component ensures long-term viability as new generation techniques emerge. Future work will explore the integration of audio-visual coherence and the model's resilience to adversarial attacks.
Summary updated