Detecting Audio-Visual Deepfakes with Fine-Grained Inconsistencies

Marcella Astrid, Enjie Ghorbel, Djamila Aouada

Introduction

As deepfake technology evolves, the realism of audio-visual forgeries has reached a point where they pose significant risks to digital security and public trust. Traditional detection methods typically focus on global features, analyzing the average characteristics of a video sequence and its corresponding audio. However, these "big picture" approaches often overlook the subtle, localized artifacts that characterize modern forgeries. This paper presents a novel detection framework that prioritizes fine-grained spatial and temporal inconsistencies, demonstrating that focusing on local details significantly improves a model's ability to generalize across different datasets.

The Limitations of Global Feature Analysis

Deepfake artifacts are rarely uniform across an entire video. Instead, they often manifest as localized discrepancies, such as a slight mismatch in lip-syncing or a momentary glitch in a specific region of a frame. When detection models rely solely on global averages, these subtle inconsistencies are frequently "washed out" by the surrounding authentic data. This leads to a lack of robustness, particularly when models are tested on datasets they were not trained on (cross-dataset evaluation). The authors argue that to build a truly effective detector, the model must be forced to examine both the spatial ("where in the frame?") and temporal ("when in the sequence?") details with high precision.

Methodology: Capturing Fine-Grained Inconsistencies

To address these challenges, the researchers proposed three primary innovations designed to capture subtle audio-visual mismatches:

1. Spatially-Local Architecture and Distance Maps

Rather than compressing an entire video frame into a single feature vector, the proposed model extracts features from numerous small spatial patches. By comparing these visual patches to the audio signal, the model generates a Distance Map. This map quantifies the level of inconsistency between the audio and each specific visual region, allowing the model to pinpoint exactly where a forgery might be occurring.

2. Cross-Attention Mechanism

In any given frame, not all regions are equally informative for detecting deepfakes. For instance, the background or a person's hair is less likely to show audio-visual inconsistencies than the mouth or eyes. The model employs a cross-attention mechanism to automatically weight these regions, focusing its analytical resources on "inconsistency-prone" areas while filtering out irrelevant spatial noise.

3. Temporally-Local Pseudo-Fake Synthesis

Effective training requires challenging data. The authors developed a new data augmentation technique called Pseudo-Fake Synthesis. Instead of creating obvious fakes by replacing entire tracks, they replace only small, random segments—perhaps a few frames or a single second of audio—with content from a different source. These "subtle" fakes force the model to become highly sensitive to minor temporal discrepancies, effectively acting as a form of adversarial training.

Experimental Evaluation

The model was evaluated using two primary benchmarks to test both its raw accuracy and its ability to generalize:

  • DFDC (DeepFake Detection Challenge): Used for in-dataset evaluation to establish a performance baseline against industry standards.
  • FakeAVCeleb: A diverse dataset used for cross-dataset evaluation, testing how the model handles forgery techniques it did not encounter during the training phase.

Results and Key Insights

The proposed method achieved state-of-the-art results, particularly in the difficult task of cross-dataset generalization.

  • In-Dataset Performance: On the DFDC dataset, the model achieved an AUC of 97.7%, significantly outperforming existing audio-visual methods such as MDS (90.7%) and AvoiD-DF (94.8%).
  • Cross-Dataset Generalization: When trained on DFDC and tested on FakeAVCeleb, the model maintained a high AUC of 84.5%. This is a notable achievement, as many high-performing models see their accuracy drop to the 50-70% range when faced with unfamiliar datasets.
  • The Importance of Locality: Ablation studies confirmed that the focus on locality is the primary driver of performance. The introduction of Pseudo-Fake training alone improved cross-dataset AUC from approximately 58% to over 82%, proving that temporal sensitivity is critical for robust detection.

Conclusion

This research demonstrates that the key to effective deepfake detection lies in the details. By shifting from global feature analysis to a fine-grained approach that utilizes spatial distance maps and temporal pseudo-fakes, the authors have created a model that is not only more accurate but also significantly more robust to new forgery types. Future work will focus on refining the distance metrics used to compare audio-visual features and optimizing the computational efficiency of the model to enable real-time detection in live environments.

Summary updated

BibTeX
@inproceedings{astrid2025detecting,
  author = {Marcella Astrid and Enjie Ghorbel and Djamila Aouada},
  title = {{Detecting Audio-Visual Deepfakes with Fine-Grained Inconsistencies}},
  booktitle = {BMVC},
  year = {2025},
  url = {https://openreview.net/forum?id=obUihhnuxE},
}
Figure