TL;DR
This paper introduces ENST, a novel dual-branch deepfake video detection model that fuses high-frequency image textures and optical flow discrepancies. By combining EfficientNet-B5 for local texture analysis and Swin Transformer for temporal flow analysis, ENST achieves superior accuracy and generalization compared to existing methods, addressing the challenge of detecting increasingly sophisticated forged videos.
What Problem Does This Paper Address?
The proliferation of deepfake technology poses significant societal risks, from misinformation to threats to personal safety. While deepfake detection methods have advanced, they often struggle with generalization to unseen data and varying compression levels. The authors identify three core reasons for this: difficulty in capturing general artifact cues, challenges in selecting appropriate models for specific feature extraction, and ineffective utilization of extracted features. This paper aims to overcome these limitations by developing a more robust and generalizable deepfake detection system.
Methods
ENST employs a dual-branch architecture:
- EfficientNet-B5 Branch: This branch focuses on extracting high-frequency and texture artifacts. It preprocesses input faces by converting RGB to YCbCr, removing luminance, and applying a Discrete Cosine Transform (DCT) to extract high-frequency components. These are then combined with Cb and Cr channels to form a 3D feature tensor. An attention module is integrated within EfficientNet-B5's MBConv layers to enhance the focus on subtle artifacts.
- Swin Transformer Branch: This branch leverages optical flow to capture temporal inconsistencies. It calculates optical flow between consecutive frames using PWC-Net and feeds this into a Swin Transformer (specifically, the first three stages, referred to as Swin Transformer-A) to capture patch embeddings.
- Feature Interaction Module: This module fuses the features from both branches. It uses 1x1 convolutions and downsampling to align feature map sizes and channel dimensions, allowing for complementary learning and enhancing global perception.
- Loss Function: For EfficientNet-B5, a novel loss function is proposed, combining Softmax loss (for global classification), ArcFace (for inter-class separation and intra-class tightness), and Single-Center Loss (SCL) (for minimizing real face distance to center and maximizing fake face distance).
Datasets
The paper uses two widely recognized datasets for training and evaluation:
- FaceForensics++: This dataset contains original videos and videos generated by four manipulation methods (Face2Face, FaceSwap, DeepFakes, NeuralTextures). It includes over 500,000 forged images across different compression levels (c0, c23, c40). The authors use c23 (medium) and c40 (high) compression levels.
- Celeb-DF (v2): This dataset comprises real and fake YouTube videos, known for its challenging segmentation and high-resolution quality. It's used to evaluate the model's generalization capabilities.
Key Insights and Ideas
- High-frequency and shallow texture are critical: The authors highlight that subtle artifacts are often more apparent in high-frequency image components and shallow texture layers, rather than deep semantic features.
- Optical flow reveals temporal inconsistencies: Real videos exhibit significant variations in optical flow, whereas deepfakes often show minimal changes, providing a strong cue for detection.
- Fusion of local and global cues is powerful: Combining the local feature extraction capabilities of CNNs (EfficientNet-B5) with the global representation power of Transformers (Swin Transformer) leads to a more comprehensive understanding of forgery artifacts.
- Enhanced loss functions improve feature discriminability: The proposed combination of Softmax, ArcFace, and SCL loss functions for EfficientNet-B5 significantly improves the ability to extract robust and distinguishable face features.
- Attention mechanisms boost artifact focus: Integrating attention modules within the EfficientNet-B5 branch helps the model concentrate on and better extract subtle artifactual information.
Key Results
- Superior Accuracy and Generalization: ENST achieves state-of-the-art performance on FaceForensics++ (e.g., 97.62% AUC and 99.30% Acc on c40) and demonstrates strong generalization capabilities on Celeb-DF (v2) (99.8% AUC).
- Robustness to Compression: The model shows excellent robustness to compressed video frames, maintaining performance even at higher compression levels (c40).
- Effectiveness of Components: Ablation studies confirm the effectiveness of the proposed loss function combination (achieving 98.67% AUC and 97.65% Acc) and the attention module, which improves accuracy when added to shallow layers of EfficientNet-B5.
Limitations and Future Work
- Attention Module Placement: While attention modules improve performance, adding them to deeper stages of EfficientNet-B5 led to decreased accuracy. The authors suggest that artifacts are concentrated in shallow layers, and further investigation into optimal attention placement is warranted.
- Computational Cost: The paper mentions that increasing attention modules can increase computational volume, implying a trade-off between performance and efficiency.
- Future Work: The paper implicitly suggests further exploration into optimal feature fusion strategies and potentially more efficient architectural designs to balance performance and computational resources. The authors also aim to further improve generalization to unseen manipulation techniques.
Summary updated