UniForensics: Face Forgery Detection via General Facial Representation

Ziyuan Fang, Hanqing Zhao, Tianyi Wei, Wenbo Zhou, Ming Wan, Zhanyi Wang, Weiming Zhang, Nenghai Yu

TL;DR

UniForensics is a novel deepfake detection framework that leverages high-level semantic facial features from a pre-trained encoder and a transformer-based video classification network. It introduces a Dynamic Video Self-Blending (DVSB) technique for data augmentation and a two-stage training approach, achieving state-of-the-art generalization and robustness against unseen forgery methods.

What Problem Does This Paper Address?

The proliferation of deepfakes poses a significant threat to digital media authenticity and privacy. Existing face forgery detection methods often rely on low-level textural features, which are vulnerable to perturbations and struggle to detect novel manipulation techniques. This leads to poor generalization capabilities, a critical issue for real-world applications. The paper aims to address this by developing a detection method that utilizes high-level semantic features and temporal inconsistencies, which are more robust and less dependent on specific forgery artifacts, thus improving generalization to unseen methods and datasets.

Methods

UniForensics combines several key innovations:

  • General Facial Representation: It utilizes a meta-functional face encoder (FaRL), pre-trained on a large dataset using image-text contrastive learning and masked image modeling, to extract rich semantic facial features (identity, geometry, expression). This encoder initializes the spatial module of a transformer-based video classification network (UniFormerV2).
  • Spatio-Temporal Modeling: The UniFormerV2 backbone, with separable temporal and spatial parameters, processes these semantic features to capture temporal inconsistencies within video sequences. This allows the model to detect anomalies in identity and motion patterns across frames.
  • Dynamic Video Self-Blending (DVSB): To generate diverse and realistic training samples, DVSB synthesizes fake videos by blending real facial videos. It introduces controlled temporal artifacts by applying a temporal filter to forgery parameters, preventing issues like frame jitter or monotonous features seen in simpler methods. It also incorporates MAdaIN (masked adaptive instance normalization) to create more challenging samples.
  • Two-Stage Training:
    1. Self-supervised Contrastive Pre-training: The model is trained on real facial videos using DVSB-generated samples. The goal is to learn generic spatio-temporal forgery traces by encouraging videos from the same forgery process to have similar representations (positive pairs).
  • Supervised Fine-tuning: The pre-trained model is then fine-tuned on a dedicated face forgery detection dataset to enhance its classification accuracy.

Datasets

  • Training (Self-supervised Pre-training): VoxCeleb2 (5994 video clips) was used for generating synthetic training data.
  • Training (Supervised Fine-tuning): FaceForensics++ (FF++) dataset, containing 1,000 original and 4,000 fake videos forged by four manipulation methods (Deepfakes, Face2Face, FaceSwap, NeuralTextures), was used. The raw version of FF++ was used for training.
  • Evaluation:
    • Cross-Dataset: Celeb-DFv2, DFDC, and FaceShifter datasets were used to evaluate generalization to unseen datasets.
  • Unseen Manipulations: FF++ was used with a leave-one-out setting to evaluate generalization to unseen manipulation methods.

Key Insights and Ideas

  • High-level semantic features are key to generalization: The paper strongly advocates for using rich, semantic facial features extracted by pre-trained encoders, rather than relying solely on low-level textures, for robust deepfake detection.
  • Temporal inconsistencies are crucial: Detecting subtle temporal inconsistencies in high-level semantic features is more effective for generalization than frame-level analysis, as many forgery methods are frame-based.
  • Controlled data synthesis is vital: Simple static or independent frame transformations for data synthesis are insufficient. DVSB's ability to generate temporally coherent yet diverse forgery artifacts is critical for training robust models.
  • Two-stage training is more effective: A self-supervised pre-training stage focused on learning general forgery traces, followed by supervised fine-tuning, significantly boosts performance and generalization compared to single-stage training.
  • Pre-trained encoders offer a powerful starting point: The success of initializing the model with FaRL highlights the transferability of general facial representation models to the specific task of deepfake detection.

Key Results

  • State-of-the-art generalization: UniForensics achieved 95.3% AUC on Celeb-DFv2 and 77.2% AUC on DFDC, outperforming existing methods.
  • Superior performance on unseen manipulations: In a leave-one-out evaluation on FF++, UniForensics achieved an average AUC of 98.8%, demonstrating strong generalization.
  • Robustness to corruptions: The model showed excellent robustness against various video corruptions (saturation, contrast, noise, compression, etc.) across different severity levels.
  • Ablation studies confirmed effectiveness: Experiments validated the contributions of the pre-trained encoder, DVSB, the two-stage training scheme, and the integration of 3D convolutions in the model architecture.

Limitations and Future Work

  • Limitations: The paper acknowledges that while DVSB generates diverse artifacts, it's still a synthetic approach. The performance on DFDC, while competitive, could still be improved.
  • Future Work: The authors suggest exploring the effectiveness of their proposed method across different spatio-temporal models and further investigating the integration of their techniques into more advanced architectures. They also plan to make their code publicly available.

Summary updated

Abstract

The rise of deepfakes has significantly heightened concerns for privacy and the authenticity of digital media, bringing widespread attention to face forgery detection. Previous deepfake detection methods mostly depend on low-level textural features vulnerable to perturbations and fall short of detecting unseen forgery methods. In contrast, high-level semantic features are less susceptible to perturbations and not limited to forgery-specific artifacts, thus having stronger generalization. Motivated by this, we propose a detection method that utilizes high-level semantic features of faces to identify inconsistencies in temporal domain. We introduce UniForensics, a novel deepfake detection framework that leverages a transformer-based video classification network, initialized with a meta-functional face encoder for enriched facial representation. In this way, we can take advantage of both the powerful spatio-temporal model and the high-level semantic information of faces. Furthermore, to leverage easily accessible real face data and guide the model in focusing on spatio-temporal features, we design a Dynamic Video Self-Blending (DVSB) method to efficiently generate training samples with diverse spatio-temporal forgery traces using real facial videos. Based on this, we advance our framework with a two-stage training approach: The first stage employs a novel self-supervised contrastive learning, where we encourage the network to focus on forgery traces by impelling videos generated by the same forgery process to have similar representations. On the basis of the representation learned in the first stage, the second stage involves fine-tuning on face forgery detection dataset to build a deepfake detector. Extensive experiments validates that UniForensics outperforms existing face forgery detection methods in generalization ability and robustness. In particular, our method achieves 95.3% and 77.2% cross dataset AUC on the challenging Celeb-DFv2 and DFDC respectively. Code will be made publicly available.
BibTeX
@article{fang2026uniforensics,
  author = {Ziyuan Fang and Hanqing Zhao and Tianyi Wei and Wenbo Zhou and Ming Wan and Zhanyi Wang and Weiming Zhang and Nenghai Yu},
  title = {{UniForensics: Face Forgery Detection via General Facial Representation}},
  journal = {IEEE Trans. Dependable Secur. Comput.},
  year = {2026},
  doi = {10.1109/tdsc.2025.3627420},
}
Figure