Detecting Face2Face Facial Reenactment in Videos

Prabhat Kumar, Mayank Vatsa, Richa Singh

Abstract

Visual content has become the primary source of information, leading to an increase in video alterations. While some are simple, sophisticated alterations like reenactment-based DeepFakes are difficult to detect. Reenactment allows a source to change target expressions, creating photo-realistic results with significant social implications. This research proposes a learning-based algorithm using a multi-stream network that learns regional artifacts to provide robust performance across various compression levels. Using a custom loss function for balanced learning, the proposed network achieves state-of-the-art classification accuracy on the FaceForensics dataset, reaching 91.20% even under heavy compression.

Introduction: The Challenge of Facial Reenactment

As DeepFake technology advances, "facial reenactment" has emerged as a significant concern. Unlike simple identity swapping, reenactment allows a source actor to manipulate a target's expressions and speech in real-time. This capability can be used maliciously to make individuals appear to say or do things they never did.

Facial reenactment involves mapping a source actor's expressions onto a target face to create realistic but fake video content.

A major hurdle for existing detection algorithms is video compression. While detectors often perform well on high-quality, uncompressed images, the heavy compression used by social media platforms like YouTube or WhatsApp "washes out" the subtle artifacts and noise patterns that these systems rely on. This research aims to create a robust detection system that maintains high performance across various compression levels.

Proposed Methodology: Multi-Stream Learning

The authors propose a multi-stream architecture based on ResNet-18 to capture both global and local inconsistencies.

1. Preprocessing and Regional Division

Faces are first detected using the S3FD algorithm. Each detected face is cropped and divided into a $2 \times 2$ grid, creating four local regions: the left eye, right eye, left cheek/mouth, and right cheek/mouth.

2. Multi-Stream Architecture

The network consists of five parallel ResNet-18 streams. One stream processes the full face, while the remaining four process the local regions. The outputs from these streams are concatenated into a 10-dimensional feature vector, which is then passed through a final fully connected layer for binary classification (Real vs. Altered).

3. Balanced Loss Function

To prevent the network from over-relying on the full-face stream, the authors introduced a custom balanced loss function ($L_{total}$). This function combines individual cross-entropy losses for each stream with a global fusion loss, forcing the regional streams to learn localized artifacts that might be overlooked in a global view.

Experimental Results

The model was evaluated using the FaceForensics dataset, which contains 1,004 videos manipulated via the Face2Face approach. Performance was tested across three H.264 compression levels: No-compression (QP 0), Easy-compression (QP 23), and Hard-compression (QP 40).

The proposed method outperformed existing benchmarks, particularly in the most challenging "hard compression" category, which simulates heavy social media processing.

Method No-Compression Easy-Compression Hard-Compression
MesoNet 96.80% 93.40% 83.20%
XceptionNet 99.93% 98.13% 87.81%
Proposed Approach 99.96% 99.10% 91.20%

Qualitative analysis using Class Activation Maps (CAMs) confirmed that the regional classifiers successfully focused on the eyes and mouth areas—regions where the Face2Face algorithm often leaves subtle artifacts during rapid movement.

Conclusion

By combining global context with localized "expert" streams, this multi-stream approach provides a robust defense against facial reenactment, even in low-quality video environments. Future work will focus on improving computational efficiency—perhaps through attention mechanisms—and testing the model's generalization across other types of generative facial manipulations.

Summary updated

Abstract

Visual content has become the primary source of information, as evident in the billions of images and videos, shared and uploaded on the Internet every single day. This has led to an increase in alterations in images and videos to make them more informative and eye-catching for the viewers worldwide. Some of these alterations are simple, like copy-move, and are easily detectable, while other sophisticated alterations like reenactment based DeepFakes are hard to detect. Reenactment alterations allow the source to change the target expressions and create photo-realistic images and videos. While technology can be potentially used for several applications, the malicious usage of automatic reenactment has a very large social implication. It is therefore important to develop detection techniques to distinguish real images and videos with the altered ones. This research proposes a learning-based algorithm for detecting reenactment based alterations. The proposed algorithm uses a multi-stream network that learns regional artifacts and provides a robust performance at various compression levels. We also propose a loss function for the balanced learning of the streams for the proposed network. The performance is evaluated on the publicly available FaceForensics dataset. The results show state-of-the-art classification accuracy of 99.96%, 99.10%, and 91.20% for no, easy, and hard compression factors, respectively.
BibTeX
@inproceedings{kumar2020detecting,
  author = {Prabhat Kumar and Mayank Vatsa and Richa Singh},
  title = {{Detecting Face2Face Facial Reenactment in Videos}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=sCJfeZvrq0w},
}
Figure