Abstract
Visual content has become the primary source of information, as evident in the billions of images and videos shared and uploaded on the Internet every single day. This has led to an increase in alterations in images and videos to make them more informative and eye-catching for the viewers worldwide. Some of these alterations are simple, like copy-move, and are easily detectable, while other sophisticated alterations like reenactment based DeepFakes are hard to detect. Reenactment alterations allow the source to change the target expressions and create photo-realistic images and videos. While the technology can be potentially used for several applications, the malicious usage of automatic reenactment has a very large social implication. It is therefore important to develop detection techniques to distinguish real images and videos with the altered ones. This research proposes a learning-based algorithm for detecting reenactment based alterations. The proposed algorithm uses a multi-stream network that learns regional artifacts and provides a robust performance at various compression levels. We also propose a loss function for the balanced learning of the streams for the proposed network. The performance is evaluated on the publicly available FaceForensics dataset. The results show state-of-the-art classification accuracy of 99.96%, 99.10%, and 91.20% for no, easy, and hard compression factors, respectively.
Introduction: The Challenge of Facial Reenactment
As DeepFake technology becomes more accessible, the "cat-and-mouse" game between creators and detectors intensifies. This paper focuses on Face2Face, a technique that transfers the expressions of a source actor to a target actor in real-time. Unlike simple image manipulations, reenactment creates photo-realistic video sequences where the target's facial movements are entirely controlled by another individual.
The primary challenge addressed in this research is video compression. Most videos shared on social media platforms like WhatsApp, YouTube, and Facebook are heavily compressed to save bandwidth. This process often smooths out the subtle pixel-level inconsistencies and "blending" artifacts that automated detectors rely on to spot a fake. While existing models perform well on high-quality, uncompressed footage, their accuracy typically plummets when faced with the "noise" and artifacts introduced by heavy compression.
Proposed Methodology: Multi-Stream Regional Analysis
The authors propose a multi-stream network designed to identify "regional artifacts" that survive the digital degradation of the compression process. The core idea is that while global facial features might be smoothed out, specific regions like the eyes and mouth retain detectable inconsistencies.
The system follows a structured pipeline:
- Preprocessing: Using the S3FD face detector, the system extracts the face and divides it into a $2 \times 2$ grid. This results in five distinct inputs: the full face, the left eye region, the right eye region, the left cheek/mouth region, and the right cheek/mouth region.
- Architecture: The model consists of five parallel ResNet-18 streams. One stream processes the full face to capture global dependencies, while the other four focus on the localized regions.
- Feature Fusion: The outputs of these five streams are concatenated into a 10-dimensional vector, which is then passed through a final fully connected layer for binary classification (Real vs. Fake).
- Balanced Loss Function: A key contribution is the custom loss function ($L_{total}$). Instead of just training the final output, the researchers calculate the cross-entropy loss for each individual stream plus a weighted fusion loss. This prevents the network from becoming biased toward the full-face stream and forces the regional streams to learn discriminative features independently.
Experimental Setup and Results
The study utilizes the FaceForensics (and FaceForensics++) dataset, which contains 1,004 unique videos from YouTube. To simulate real-world conditions, the model was evaluated across three H.264 compression levels: No-compression (raw), Easy-compression (QP 23), and Hard-compression (QP 40).
The proposed method outperformed several baselines, including MesoNet and XceptionNet, particularly in high-compression scenarios. The results demonstrate a significant leap in robustness:
| Compression Level | Accuracy |
|---|---|
| No Compression | 99.96% |
| Easy Compression | 99.10% |
| Hard Compression | 91.20% |
In the "Hard" compression category, the proposed model's 91.20% accuracy is a significant improvement over the 83.20% achieved by MesoNet and the 87.81% by XceptionNet.
Key Insights
Several critical insights emerged from the study:
- Resilience of Local Artifacts: While compression obscures global artifacts like subtle skin texture mismatches, it struggles to hide inconsistencies in high-movement areas like the mouth and eyes.
- Shift in Attention: Class activation maps (CAMs) revealed that in uncompressed videos, the model focuses on the nose and mouth. However, as compression increases, the model shifts its "attention" to the eye regions, suggesting that eye-area artifacts are more resilient to digital degradation.
- The Power of Balanced Learning: Without the custom loss function, the model tends to over-rely on the full-face stream. Forcing each regional stream to contribute independently makes the system much more robust.
Conclusion and Future Directions
While the multi-stream approach provides state-of-the-art results, it comes with increased computational overhead due to the five parallel ResNet-18 streams. Future work will focus on reducing model complexity and exploring how well this regional approach generalizes to other DeepFake methods, such as GAN-based synthesis or identity swapping. Additionally, the authors suggest incorporating attention mechanisms to automatically learn dependencies between facial regions rather than relying on a fixed grid.
Summary updated