Watch Those Words: Video Falsification Detection Using Word-Conditioned Facial Motion

Shruti Agarwal, Liwen Hu, Evonne Ng, Trevor Darrell, Hao Li, Anna Rohrbach

Abstract

As AI-driven video synthesis reaches perceptual realism, traditional detection methods based on visual artifacts are becoming less effective. This paper introduces a novel forensic technique that leverages "semantic biometrics"—the unique, person-specific facial and head movements associated with specific spoken words. By analyzing word-conditioned facial motion, the proposed system can detect high-tech deepfakes as well as "cheapfakes" like audio dubbing and impersonators. Our method achieves an average AUC of 0.92 to 0.97 across various falsification scenarios, significantly outperforming state-of-the-art visual detectors in identifying non-digitally manipulated content.

1. Introduction

The proliferation of deepfakes and "cheapfakes" poses a significant threat to information integrity. Traditional forensic methods often rely on detecting digital artifacts—glitches or inconsistencies in the pixel data. However, as synthesis technology improves, these artifacts vanish. Furthermore, "cheapfakes," such as videos featuring impersonators or simple audio dubbing, contain no digital manipulation to detect.

This research, a collaboration between UC Berkeley and Pinscreen, proposes a shift toward "semantic biometrics." The fundamental premise is that individuals possess unique behavioral "signatures"—idiosyncratic facial muscle movements and head poses—that are consistently triggered by specific spoken words. By verifying these patterns, we can authenticate the identity of the person in the video regardless of the synthesis method used.

2. Methodology: Word-Conditioned Facial Motion Analysis

The proposed system follows a structured pipeline to extract and classify behavioral biometrics:

  1. Transcription and Temporal Alignment: Using the DeepSpeech architecture, the system transcribes the audio track and identifies the precise start and end frames for every word spoken.
  2. Feature Extraction: To ensure interpretability and robustness, the system avoids "black box" features. Instead, it extracts Action Units (AUs)—standardized markers of facial muscle activity—and 3D head pose data (rotation and translation).
  3. Feature Engineering: For each word occurrence, the system calculates the "range of motion" (the delta between maximum and minimum values) for AUs and head pose within the word's temporal window. This approach mitigates the impact of minor timing misalignments.
  4. Classification: Person-specific linear logistic regression models are trained for individual words (e.g., a specific classifier for Barack Obama saying "everybody").
Figure 4: Sequential frames of a subject speaking, highlighting the dynamic facial expressions and head movements that form the basis of the biometric signature.

3. Experimental Results

The method was rigorously tested using the World Leaders Dataset (WLDR), featuring figures like Obama, Trump, Biden, and Harris, alongside talk-show hosts. The falsification scenarios included state-of-the-art lip-syncing (Wav2Lip), FaceSwaps, professional impersonators, and audio dubbing.

3.1 Performance and Generalization

The system achieved an average AUC between 0.92 and 0.97 across all test scenarios. A key finding was the superiority of word-level "anchors" over phoneme-level analysis; words were found to trigger more repeatable and distinct semantic gestures.

3.2 Detecting "Cheapfakes"

One of the most significant results was the system's performance on dubbed videos. While state-of-the-art visual detectors like XceptionNet and LipForensics failed completely (AUC ~0.50) because the visual data remained "real," the word-conditioned approach maintained an AUC of 0.92. It successfully identified the lack of correlation between the spoken words and the subject's established biometric patterns.

Figure 2: Distribution of authenticity scores for real (blue) and fake (orange) video samples. The distinct separation in the primary plot illustrates the high discriminative power of the word-conditioned biometric model compared to non-conditioned baselines.

4. Discussion and Limitations

The study demonstrates that even skilled actors struggle to replicate the micro-gestures associated with a target's specific vocabulary. However, the method has limitations: it requires approximately one hour of high-quality training footage, making it primarily applicable to public figures. Additionally, the system's accuracy depends on reliable 3D facial tracking, which can be challenged by extreme head poses or poor lighting.

5. Conclusion

By focusing on what a person does rather than how a video is rendered, word-conditioned facial motion analysis provides a powerful new layer of defense in the fight against misinformation. Future work will explore denser 3D features and multi-lingual applications to further enhance the robustness of semantic forensics.

Summary updated

Abstract

In today's era of digital misinformation, we are increasingly faced with new threats posed by video falsification techniques. Such falsifications range from cheapfakes (e.g., lookalikes or audio dubbing) to deepfakes (e.g., sophisticated AI media synthesis methods), which are becoming perceptually indistinguishable from real videos. To tackle this challenge, we propose a multi-modal semantic forensic approach to discover clues that go beyond detecting discrepancies in visual quality, thereby handling both simpler cheapfakes and visually persuasive deepfakes. In this work, our goal is to verify that the purported person seen in the video is indeed themselves by detecting anomalous facial movements corresponding to the spoken words. We leverage the idea of attribution to learn person-specific biometric patterns that distinguish a given speaker from others. We use interpretable Action Units (AUs) to capture a person's face and head movement as opposed to deep CNN features, and we are the first to use word-conditioned facial motion analysis. We further demonstrate our method's effectiveness on a range of fakes not seen in training including those without video manipulation, that were not addressed in prior work.
BibTeX
@inproceedings{agarwal2023watch,
  author = {Shruti Agarwal and Liwen Hu and Evonne Ng and Trevor Darrell and Hao Li and Anna Rohrbach},
  title = {{Watch Those Words: Video Falsification Detection Using Word-Conditioned Facial Motion}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00469},
}
Figure