Cross-Domain Local Characteristic Enhanced Deepfake Video Detection

Zihan Liu, Hanyi Wang, Shilin Wang

Abstract

As ultra-realistic face forgery techniques emerge, deepfake detection has attracted increasing attention due to security concerns. Many detectors cannot achieve accurate results when detecting unseen manipulations despite excellent performance on known forgeries. In this paper, we are motivated by the observation that the discrepancies between real and fake videos are extremely subtle and localized, and inconsistencies or irregularities can exist in some critical facial regions across various information domains. To this end, we propose a novel pipeline, Cross-Domain Local Forensics (XDLF), for more general deepfake video detection. In the proposed pipeline, a specialized framework is presented to simultaneously exploit local forgery patterns from space, frequency, and time domains, thus learning cross-domain features to detect forgeries. Moreover, the framework leverages four high-level forgery-sensitive local regions of a human face to guide the model to enhance subtle artifacts and localize potential anomalies. Extensive experiments on several benchmark datasets demonstrate the impressive performance of our method, and we achieve superiority over several state-of-the-art methods on cross-dataset generalization.

Introduction

The core challenge in modern deepfake detection is generalization. While many detectors perform brilliantly on datasets they have been trained on, they often fail when faced with unseen manipulation techniques. This occurs because many models overfit to specific digital artifacts—such as blending boundaries—rather than learning the fundamental irregularities inherent in synthetic videos.

Research indicates that while global facial structures may appear perfect, deepfake discrepancies are typically subtle, localized, and hidden across different information domains. For instance, a forgery might be spatially convincing but exhibit anomalies in its high-frequency noise or temporal consistency. To address this, we introduce the Cross-Domain Local Forensics (XDLF) pipeline, which aggregates clues from the spatial, frequency, and temporal domains to improve detection robustness.

Methodology

The XDLF framework is built on a two-stream 3D CNN architecture designed to mine forgery patterns from three domains: space, frequency, and time.

The overall pipeline of Cross-Domain Local Forensics (XDLF), illustrating the flow from preprocessing to classification.

The methodology consists of four primary components:

  1. Frequency Decomposition: Instead of relying solely on RGB pixels, the model processes frequency maps generated via Discrete Cosine Transform (DCT). This amplifies subtle manipulation traces that are often invisible to the human eye.
  2. Forgery-Sensitive Local Regions (FSLR): The system identifies four key areas—the left eye, right eye, nose, and mouth—as being particularly difficult for generative models to render consistently.
  3. FSLR-Guided Feature Enhancement (FGFE): This novel attention mechanism uses the bounding boxes of the eyes, nose, and mouth to guide the neural network, forcing it to focus on local textural artifacts in these high-stakes regions.
  4. Cross-Domain Fusion: As shown in the detailed architecture below, the model utilizes Cross Attention and Feature Fusion modules to allow the RGB and Frequency streams to interact at multiple levels (low, mid, and high), ensuring the final classification is based on a holistic view of the data.
Detailed architecture of the Cross-Domain Local Forensics module, highlighting the interaction between RGB and frequency streams.

Experimental Results

The researchers evaluated XDLF on industry-standard benchmarks, including FaceForensics++ (FF++), Celeb-DF (CDF), and the DeepFake Detection Challenge (DFDC).

Performance and Generalization

XDLF achieved impressive results, particularly in cross-dataset generalization:

  • In-Dataset (FF++): Achieved a 99.7% AUC on high-quality videos, matching top-tier state-of-the-art (SOTA) methods.
  • Cross-Dataset (Celeb-DF): Reached 82.6% AUC, surpassing SOTA methods like LipForensics (82.4%).
  • Cross-Dataset (DFDC): Reached 73.8% AUC, further demonstrating its ability to handle unseen manipulations.

Key Insights

Ablation studies highlighted that frequency data is often more robust than color for generalization, as synthesis noise remains a more consistent giveaway than algorithm-specific color artifacts. Furthermore, the transition to 3D spatio-temporal CNNs confirmed that temporal inconsistencies (frame-to-frame jitter) are a primary weakness of current deepfake generators. Finally, explicitly guiding the model to focus on the eyes, nose, and mouth improved AUC on Celeb-DF by over 3%.

Conclusion

The XDLF pipeline demonstrates that exploiting cross-domain local characteristics is a highly effective strategy for developing generalizable deepfake detectors. By combining spatial, frequency, and temporal analysis with semantic guidance in forgery-sensitive regions, the model achieves high performance without the need for massive external pre-training. Future work may explore additional facial regions or alternative frequency representations to counter increasingly sophisticated forgery techniques.

Summary updated

Abstract

As ultra-realistic face forgery techniques emerge, deepfake detection has attracted increasing attention due to security concerns. Many detectors cannot achieve accurate results when detecting unseen manipulations despite excellent performance on known forgeries. In this paper, we are motivated by the observation that the discrepancies between real and fake videos are extremely subtle and localized, and inconsistencies or irregularities can exist in some critical facial regions across various information domains. To this end, we propose a novel pipeline, Cross-Domain Local Forensics (XDLF), for more general deepfake video detection. In the proposed pipeline, a specialized framework is presented to simultaneously exploit local forgery patterns from space, frequency, and time domains, thus learning cross-domain features to detect forgeries. Moreover, the framework leverages four high-level forgery-sensitive local regions of a human face to guide the model to enhance subtle artifacts and localize potential anomalies. Extensive experiments on several benchmark datasets demonstrate the impressive performance of our method, and we achieve superiority over several state-of-the-art methods on cross-dataset generalization. We also examined the factors that contribute to its performance through ablations, which suggests that exploiting cross-domain local characteristics is a noteworthy direction for developing more general deepfake detectors.
BibTeX
@inproceedings{liu2022cross,
  author = {Zihan Liu and Hanyi Wang and Shilin Wang},
  title = {{Cross-Domain Local Characteristic Enhanced Deepfake Video Detection}},
  booktitle = {ACCV},
  year = {2022},
  url = {https://openreview.net/forum?id=WVC8FZ0D1-},
}
Figure