Video Forgery Detection Using Multiple Cues on Fusion of EfficientNet and Swin Transformer

Chenyu Liu, Jia Li, Junxian Duan, Huaibo HuangICLR 2022 · OpenReview

TL;DR

This paper introduces ENST, a novel dual-branch deepfake video detection model that fuses high-frequency image textures and optical flow discrepancies. By combining EfficientNet-B5 for local texture analysis and Swin Transformer for temporal flow analysis, ENST achieves superior accuracy and generalization compared to existing methods, addressing the challenge of detecting increasingly sophisticated forged videos.

What Problem Does This Paper Address?

The proliferation of deepfake technology poses significant societal risks, from misinformation to threats to personal safety. While deepfake detection methods have advanced, they often struggle with generalization to unseen data and varying compression levels. The authors identify three core reasons for this: difficulty in capturing general artifact cues, challenges in selecting appropriate models for specific feature extraction, and ineffective utilization of extracted features. This paper aims to overcome these limitations by developing a more robust and generalizable deepfake detection system.

Methods

ENST employs a dual-branch architecture:

  • EfficientNet-B5 Branch: This branch focuses on extracting high-frequency and texture artifacts. It preprocesses input faces by converting RGB to YCbCr, removing luminance, and applying a Discrete Cosine Transform (DCT) to extract high-frequency components. These are then combined with Cb and Cr channels to form a 3D feature tensor. An attention module is integrated within EfficientNet-B5's MBConv layers to enhance the focus on subtle artifacts.
  • Swin Transformer Branch: This branch leverages optical flow to capture temporal inconsistencies. It calculates optical flow between consecutive frames using PWC-Net and feeds this into a Swin Transformer (specifically, the first three stages, referred to as Swin Transformer-A) to capture patch embeddings.
  • Feature Interaction Module: This module fuses the features from both branches. It uses 1x1 convolutions and downsampling to align feature map sizes and channel dimensions, allowing for complementary learning and enhancing global perception.
  • Loss Function: For EfficientNet-B5, a novel loss function is proposed, combining Softmax loss (for global classification), ArcFace (for inter-class separation and intra-class tightness), and Single-Center Loss (SCL) (for minimizing real face distance to center and maximizing fake face distance).

Datasets

The paper uses two widely recognized datasets for training and evaluation:

  • FaceForensics++: This dataset contains original videos and videos generated by four manipulation methods (Face2Face, FaceSwap, DeepFakes, NeuralTextures). It includes over 500,000 forged images across different compression levels (c0, c23, c40). The authors use c23 (medium) and c40 (high) compression levels.
  • Celeb-DF (v2): This dataset comprises real and fake YouTube videos, known for its challenging segmentation and high-resolution quality. It's used to evaluate the model's generalization capabilities.

Key Insights and Ideas

  • High-frequency and shallow texture are critical: The authors highlight that subtle artifacts are often more apparent in high-frequency image components and shallow texture layers, rather than deep semantic features.
  • Optical flow reveals temporal inconsistencies: Real videos exhibit significant variations in optical flow, whereas deepfakes often show minimal changes, providing a strong cue for detection.
  • Fusion of local and global cues is powerful: Combining the local feature extraction capabilities of CNNs (EfficientNet-B5) with the global representation power of Transformers (Swin Transformer) leads to a more comprehensive understanding of forgery artifacts.
  • Enhanced loss functions improve feature discriminability: The proposed combination of Softmax, ArcFace, and SCL loss functions for EfficientNet-B5 significantly improves the ability to extract robust and distinguishable face features.
  • Attention mechanisms boost artifact focus: Integrating attention modules within the EfficientNet-B5 branch helps the model concentrate on and better extract subtle artifactual information.

Key Results

  • Superior Accuracy and Generalization: ENST achieves state-of-the-art performance on FaceForensics++ (e.g., 97.62% AUC and 99.30% Acc on c40) and demonstrates strong generalization capabilities on Celeb-DF (v2) (99.8% AUC).
  • Robustness to Compression: The model shows excellent robustness to compressed video frames, maintaining performance even at higher compression levels (c40).
  • Effectiveness of Components: Ablation studies confirm the effectiveness of the proposed loss function combination (achieving 98.67% AUC and 97.65% Acc) and the attention module, which improves accuracy when added to shallow layers of EfficientNet-B5.

Limitations and Future Work

  • Attention Module Placement: While attention modules improve performance, adding them to deeper stages of EfficientNet-B5 led to decreased accuracy. The authors suggest that artifacts are concentrated in shallow layers, and further investigation into optimal attention placement is warranted.
  • Computational Cost: The paper mentions that increasing attention modules can increase computational volume, implying a trade-off between performance and efficiency.
  • Future Work: The paper implicitly suggests further exploration into optimal feature fusion strategies and potentially more efficient architectural designs to balance performance and computational resources. The authors also aim to further improve generalization to unseen manipulation techniques.

Summary updated

Abstract

The rapid development of video processing technology makes it easy for people to forge videos without leaving visual artifacts. The spread of forged videos may lead to moral and legal consequences and pose a potential threat to people's lives and social stability. So it is significant to identify deepfake video information. Although the previous detection methods have achieved high accuracy, the generalization is poor when facing unprecedented data in the real scene. There are three fundamental reasons. The first is that capturing the general clue of artifacts is difficult. The second is that selecting the appropriate model is challenging in specific feature extraction. The third is that exploiting fully and effectively the extracted features is hard. We find that the high-frequency information in the image and the texture in the shallow layer of the model expose the subtle artifacts. The optical flow of the real video has variations while the optical flow of the deepfake video has rarely variations. Furthermore, consecutive frames in the real video have temporal consistency. In this paper, we propose a dual-branch video forgery detection model named ENST, which integrates parallelly and interactively EfficientNet-B5 and Swin Transformer. Specifically, EfficientNet-B5 extracts the artifacts information of high frequency and texture in the shallow layer of the model. Swin Transformer captures the subtle discrepancies between optical flows. To extract more robust face features, we design a new loss function for EfficientNet-B5. In addition, we also introduce the attention mechanism into EfficientNet-B5 to enhance the extracted features. We conduct test experiments on FaceForensics++ and Celeb-DF (v2) datasets, and comprehensive results show that ENST has higher accuracy and generalization, which is superior to the most advanced methods.
BibTeX
@inproceedings{liu2022video,
  author = {Chenyu Liu and Jia Li and Junxian Duan and Huaibo Huang},
  title = {{Video Forgery Detection Using Multiple Cues on Fusion of EfficientNet and Swin Transformer}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=K3uRhaKJuZg},
}
Figure