Not made for each other- Audio-Visual Dissonance-based Deepfake Detection and Localization

Komal Chugh, Parul Gupta, Abhinav Dhall, Ramanathan Subramanian

TL;DR

This paper introduces a novel deepfake detection method that leverages "modality dissonance" – the mismatch between audio and visual cues in a video. By training a neural network to identify these inconsistencies, the authors achieve state-of-the-art results and demonstrate the ability to pinpoint the exact segments of a video that have been manipulated.

What Problem Does This Paper Address?

The proliferation of deepfake technology presents a significant challenge, as AI-generated videos are becoming increasingly indistinguishable from real ones. While many existing deepfake detection methods focus solely on visual artifacts, this paper acknowledges that manipulations can occur in either the audio or visual channel, or both. The core problem is to develop a robust detection system that can identify these sophisticated fakes, even when the visual authenticity is high, by exploiting the inherent disharmony that arises when audio and visual streams are not perfectly synchronized or congruent.

Methods

The authors propose a Modality Dissonance Score (MDS), calculated by comparing audio and visual features extracted from short, 1-second video segments. Their approach uses a bi-stream neural network: one stream processes visual data (face crops) and the other processes audio data (MFCC features).

Key methodological contributions include:

  • Contrastive Loss for Modality Dissonance: A contrastive loss function is employed to explicitly encourage higher dissimilarity between audio-visual features for fake videos and lower dissimilarity for real videos. This is the core mechanism for capturing "modality dissonance."
  • Unimodal Cross-Entropy Losses: In addition to the contrastive loss, unimodal cross-entropy losses are applied to both the audio and visual streams. This allows the network to learn discriminative features independently for each modality, making the system more robust even if one modality is missing or corrupted.
  • Chunk-wise Analysis: Instead of processing entire videos, the method operates on 1-second segments. This fine-grained approach not only improves detection accuracy but also enables temporal forgery localization.
  • Feature Learning: Discriminative features are learned for both audio and visual channels in a chunk-wise manner.

Datasets

The paper evaluates its method on two public audio-visual deepfake datasets:

  • Deepfake-TIMIT: This dataset contains videos with manipulated facial expressions and speech.
  • DFDC (Deepfake Detection Challenge) Dataset: A larger dataset with a variety of manipulations, including face swapping. The authors used a subset of 18,000 videos for their experiments.

Key Insights and Ideas

  • Audio-Visual Dissonance is a Powerful Cue: The fundamental insight is that even highly realistic visual deepfakes often exhibit subtle (or not-so-subtle) mismatches with their accompanying audio, and this "dissonance" is a strong indicator of manipulation.
  • Chunking is Key for Localization and Accuracy: Analyzing short segments (1 second) rather than whole videos allows for more precise detection of transient forgeries and improves overall performance. This is a significant departure from methods that aggregate features over longer durations.
  • Unimodal Learning Enhances Robustness: Including cross-entropy losses for individual audio and visual streams makes the model less reliant on the presence of both modalities and improves the discriminative power of the learned features.
  • Temporal Localization is Achievable: The chunk-wise approach naturally lends itself to identifying where in a video the manipulation occurs, not just if it's fake.

Key Results

  • State-of-the-Art Performance: The proposed MDS-based method achieves state-of-the-art results on the DFDC dataset, improving the AUC score by up to 7% compared to previous methods.
  • High Accuracy on DFDC: The combined loss function achieved an AUC of 91.7% on the DFDC dataset (with a specific subset), and 90.55% on the full DFDC dataset.
  • Competitive Performance on DFTIMIT: The method also performs competitively on the DFTIMIT dataset, achieving AUCs of 97.9% (LQ) and 96.8% (HQ).
  • Temporal Forgery Localization Demonstrated: The paper shows successful localization of manipulated segments in synthesized videos, visually highlighting the dissonant segments.
  • Ablation Studies Highlight Contributions: Experiments show that the combined loss (contrastive + cross-entropy) significantly outperforms using only contrastive loss (91.7% vs. 86.1% AUC on a subset of DFDC). Individual stream performance was also analyzed, with the visual stream showing higher discriminative ability (89.7% AUC) than the audio stream (50.0% AUC) on DFDC, likely due to the nature of manipulations in that dataset.

Limitations and Future Work

  • Limitations:
    • The paper notes that the DFDC dataset primarily contains visual manipulations, which might explain the lower performance of the audio-only stream.
  • The authors mention a discrepancy in the train-test split of a subset of the DFDC dataset used in a prior work, which they corrected for their comparison.
  • The current method relies on face detection, which could be a bottleneck if faces are not clearly visible or are occluded.
  • Future Work:
    • Human Assessment Integration: Incorporating human perception data (e.g., EEG, eye-gaze) alongside content analysis.
  • Multiple Instance Learning: Exploring MIL for improved transient forgery detection.
  • Real-time Detection: Developing algorithms for real-time forgery detection by processing audio-visual information over 1-second segments.

Summary updated

Abstract

We propose detection of deepfake videos based on the dissimilarity between the audio and visual modalities, termed as the Modality Dissonance Score (MDS). We hypothesize that manipulation of either modality will lead to dis-harmony between the two modalities, eg, loss of lip-sync, unnatural facial and lip movements, etc. MDS is computed as an aggregate of dissimilarity scores between audio and visual segments in a video. Discriminative features are learnt for the audio and visual channels in a chunk-wise manner, employing the cross-entropy loss for individual modalities, and a contrastive loss that models inter-modality similarity. Extensive experiments on the DFDC and DeepFake-TIMIT Datasets show that our approach outperforms the state-of-the-art by up to 7%. We also demonstrate temporal forgery localization, and show how our technique identifies the manipulated video segments.
BibTeX
@inproceedings{chugh2020not,
  author = {Komal Chugh and Parul Gupta and Abhinav Dhall and Ramanathan Subramanian},
  title = {{Not made for each other- Audio-Visual Dissonance-based Deepfake Detection and Localization}},
  booktitle = {ACM Multimedia},
  year = {2020},
  url = {https://openreview.net/forum?id=qPSSWsksD-},
}
Figure