Collaborative Feature Learning for Fine-grained Facial Forgery Detection and Segmentation

Weinan Guan, Wei Wang, Jing Dong, Bo Peng, Tieniu TanarXiv 2023 · OpenReview

Abstract

Detecting maliciously falsified facial images and videos has attracted extensive attention from digital-forensics and computer-vision communities. An important topic in manipulation detection is the localization of the fake regions. Previous work related to forgery detection mostly focuses on the entire faces. However, recent forgery methods have developed to edit important facial components while maintaining others unchanged. This drives us to not only focus on the forgery detection but also fine-grained falsified region segmentation. In this paper, we propose a collaborative feature learning approach to simultaneously detect manipulation and segment the falsified components. With the collaborative manner, detection and segmentation can boost each other efficiently. To enable our study of forgery detection and segmentation, we build a facial forgery dataset consisting of both entire and partial face forgeries with their pixel-level manipulation ground-truth. Experiment results have justified the mutual promotion between forgery detection and manipulated region segmentation. The overall performance of the proposed approach is better than the state-of-the-art detection or segmentation approaches. The visualization results have shown that our proposed model always captures the artifacts on facial regions, which is more reasonable.

Introduction

Most existing deepfake detection methods treat the problem as a simple binary classification: is the entire face real or fake? However, modern forgery techniques have evolved beyond simple "face-swapping." Newer methods, like R-Face, can precisely edit individual facial components—such as just the eyes, nose, or mouth—while leaving the rest of the face untouched.

Comparison of different forgery techniques (FaceSwap, Deepfakes, R-Face) and their corresponding segmentation masks.

Standard detectors often struggle with these "fine-grained" forgeries because the global features of the face remain largely authentic. Furthermore, black-box classifiers offer no explanation for their decisions. This paper addresses the need for explainable detection by forcing the model to localize the specific artifacts that lead to its "fake" verdict.

Collaborative Forgery Feature Learning

The proposed architecture utilizes a Collaborative Forgery Feature Learning approach. The core idea is that detection and segmentation are complementary tasks that should "boost" each other through a shared representation.

  • Shared Encoder: The model uses a shared XceptionNet backbone to extract features. This ensures that the latent representation contains both high-level semantic information (for classification) and low-level spatial details (for segmentation).
  • Dual Branches: The model splits into a Detection Branch (using Global Average Pooling for a probability score) and a Segmentation Branch (using a deconvolutional decoder to produce a pixel-level mask).
  • Collaborative Loss: The network is trained using a joint loss function—a weighted sum of Binary Cross-Entropy (BCE) for both detection and segmentation.
  • Mutual Promotion: During training, the segmentation task forces the encoder to focus on local artifacts, while the detection task provides a global signal that helps the segmenter ignore natural facial features that might look like artifacts.

Dataset Construction

To enable this study, the authors constructed a new dataset containing 30,000 falsified images with pixel-level ground-truth masks. This includes Entire Face Forgeries sourced from FaceForensics++ (FaceSwap and Deepfakes) and Partial Face Forgeries generated using R-Face to edit specific components on images from the CelebAMask-HQ dataset.

Examples of source images, manipulated images, and the corresponding ground-truth masks for various facial components like glasses and facial features.

Key Results and Insights

The proposed "Ours (Xcep)" model outperformed several baselines, including standalone XceptionNet, MesoNet, and U-Net.

  • Detection Accuracy: Achieved 99.10% overall accuracy, surpassing the standard XceptionNet (98.61%).
  • Segmentation Performance: Achieved an Intersection over Union (IoU) of 0.9659, significantly outperforming general-purpose segmentation networks like U-Net (0.8678).
  • Segmentation as Attention: The segmentation task acts as a supervised attention mechanism. It forces the shared encoder to "look" at the forged pixels, which naturally improves the accuracy of the detection branch.
  • Partial vs. Full Forgery Dynamics: Interestingly, segmentation performance (IoU) was often higher on partially edited faces than on full face swaps, suggesting that fine-grained edits leave more distinct, localized boundaries.
  • Improved Interpretability: Visualization using Grad-CAM++ demonstrated that the collaborative model focuses strictly on facial regions with artifacts, whereas standard models often get distracted by background elements.

Conclusion

This paper introduces a collaborative multi-task learning framework that simultaneously detects facial forgeries and segments the specific modified regions at a pixel level. By sharing a feature encoder between detection and segmentation tasks, the model achieves state-of-the-art performance, proving that knowing where a face is fake significantly helps in deciding if it is fake. Future work will focus on extending this approach to video sequences and improving generalization against unseen forgery techniques.

Summary updated

Abstract

Detecting maliciously falsified facial images and videos has attracted extensive attention from digital-forensics and computer-vision communities. An important topic in manipulation detection is the localization of the fake regions. Previous work related to forgery detection mostly focuses on the entire faces. However, recent forgery methods have developed to edit important facial components while maintaining others unchanged. This drives us to not only focus on the forgery detection but also fine-grained falsified region segmentation. In this paper, we propose a collaborative feature learning approach to simultaneously detect manipulation and segment the falsified components. With the collaborative manner, detection and segmentation can boost each other efficiently. To enable our study of forgery detection and segmentation, we build a facial forgery dataset consisting of both entire and partial face forgeries with their pixel-level manipulation ground-truth. Experiment results have justified the mutual promotion between forgery detection and manipulated region segmentation. The overall performance of the proposed approach is better than the state-of-the-art detection or segmentation approaches. The visualization results have shown that our proposed model always captures the artifacts on facial regions, which is more reasonable.
BibTeX
@misc{guan2023collaborative,
  author = {Weinan Guan and Wei Wang and Jing Dong and Bo Peng and Tieniu Tan},
  title = {{Collaborative Feature Learning for Fine-grained Facial Forgery Detection and Segmentation}},
  year = {2023},
  eprint = {2304.08078},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.08078},
}
Figure