TL;DR
This paper introduces TIMIT-TTS, a novel multimodal audio-visual deepfake dataset designed to address the growing challenge of detecting sophisticated synthetic media. By leveraging Text-to-Speech (TTS) and Dynamic Time Warping (DTW), the authors generate realistic synthetic audio tracks to accompany existing deepfake videos, enabling the development and evaluation of cross-modal forensic detectors.
What Problem Does This Paper Address?
The proliferation of deep learning has made it increasingly easy to create highly realistic synthetic media, including videos where both the visual and audio components are manipulated (deepfakes). While the multimedia forensics community has developed detectors to identify these fakes, most existing tools operate on a single modality (either audio or video) at a time. This monomodal approach is insufficient for detecting advanced deepfakes that exploit inconsistencies between audio and visual cues. A key bottleneck in developing multimodal detectors is the scarcity of datasets containing forgeries where both audio and video are manipulated. Existing datasets often focus on only one falsified modality, leaving a critical gap for training and testing robust multimodal forensic systems.
Methods
The core contribution is a pipeline for generating multimodal deepfake audio. This pipeline takes a video (V) as input, which consists of an audio track (x) and a visual component (y).
- Speech-to-Text: The original audio track (x) is transcribed into text (T).
- Text-to-Speech (TTS) Synthesis: A chosen TTS system generates a synthetic audio track (s) from the text (T). The paper utilizes 12 state-of-the-art TTS models, including popular ones like Tacotron, Tacotron2, GlowTTS, FastSpeech2, and VITS, to ensure a diverse range of synthetic speech characteristics.
- Post-processing: To enhance realism and challenge detectors, two post-processing steps are applied:
- Audio-Video Synchronization (DTW): Dynamic Time Warping is used to align the synthetic audio track (s) with the original audio track (x) to ensure temporal coherence with the video. This is crucial because asynchronous audio and video are easily detectable.
- Data Augmentation: Techniques like Gaussian noise injection, pitch shifting, and MP3 compression are applied to further obscure synthetic artifacts and simulate real-world degradation.
The resulting synthetic audio track (î) is then combined with the original video component (y) to create a multimodal deepfake.
Datasets
- VidTIMIT: This dataset serves as the reference for real audio-video recordings. It contains 430 videos of 43 individuals speaking short sentences. The authors chose VidTIMIT because it provides official transcripts, eliminating the need for error-prone automatic transcription.
- DeepfakeTIMIT: This dataset, previously released, contains video deepfakes derived from VidTIMIT.
- TIMIT-TTS (New Dataset): This is the primary contribution of the paper. It's a novel multimodal deepfake dataset generated by applying the described pipeline to VidTIMIT and DeepfakeTIMIT. It comprises approximately 80,000 synthetic audio tracks, created using 12 different TTS generators, and is released in four versions: clean (no post-processing), DTW-synchronized, augmented, and DTW-synchronized + augmented.
Key Insights and Ideas
- The Necessity of Multimodal Data: The experiments clearly demonstrate that monomodal detectors struggle with post-processed deepfakes, and multimodal analysis offers a significant advantage.
- DTW's Impact on Detectability: While DTW synchronization makes audio more aligned with video, it paradoxically makes the synthetic audio harder to detect by audio-only detectors, highlighting the need for detectors trained on such synchronized data.
- Speaker Identity as a Distractor: When TTS models are trained on a single speaker (like LJSpeech), detectors tend to rely on speaker identity rather than synthetic artifacts for classification. This suggests that datasets with diverse speakers across multiple TTS models are crucial for developing robust attribution methods that focus on generator-specific traces.
- Real-world Data is Messy: The significant drop in detection performance when dealing with compressed video (QP=40) underscores the importance of developing detectors robust to real-world degradation.
Key Results
- Audio Binary Classification: The AUC for detecting deepfakes dropped from 0.98 (clean) to 0.77 (DTW + augmented), indicating that post-processing significantly challenges monomodal audio detectors.
- Audio Attribution: Identifying the specific TTS generator used for synthesis was more challenging for models trained on a single speaker (LJSpeech) compared to multi-speaker models. Balanced accuracy dropped from 0.77 to 0.67 for single-speaker models.
- Video Binary Classification: Detection performance also degraded with compression, with AUC dropping from 0.99 (high quality) to 0.79 (low quality).
- Multimodal Classification: Multimodal approaches consistently outperformed monomodal ones. In Scenario 2 (mixed real/fake modalities), multimodal detection achieved an AUC improvement of 0.15 over single modalities, even with simple score fusion.
Limitations and Future Work
- Limited TTS Models: The dataset currently includes 12 TTS models. Future work could incorporate more models and explore the impact of different vocoders.
- Limited Speaker Diversity: While TIMIT-TTS offers multiple speakers, the authors acknowledge that some TTS models are limited to the LJSpeech speaker. Future versions could aim for more speakers across all models.
- Exclusion of Voice Conversion (VC): The current pipeline focuses on TTS. Future work plans to include VC methods to generate a more comprehensive dataset.
- Simple Fusion Methods: The multimodal fusion used in experiments was basic (averaging or taking the max score). More sophisticated fusion techniques could further improve detection performance.
- Need for More Data: The authors emphasize that while TIMIT-TTS is a significant step, more multimodal deepfake datasets are needed to train and test robust detectors for in-the-wild conditions.
Summary updated