TIMIT-TTS: A Text-to-Speech Dataset for Multimodal Synthetic Media Detection

Davide Salvi, Brian C. Hosler, Paolo Bestagini, Matthew C. Stamm, Stefano Tubaro

TL;DR

This paper introduces TIMIT-TTS, a novel multimodal audio-visual deepfake dataset designed to address the growing challenge of detecting sophisticated synthetic media. By leveraging Text-to-Speech (TTS) and Dynamic Time Warping (DTW), the authors generate realistic synthetic audio tracks to accompany existing deepfake videos, enabling the development and evaluation of cross-modal forensic detectors.

What Problem Does This Paper Address?

The proliferation of deep learning has made it increasingly easy to create highly realistic synthetic media, including videos where both the visual and audio components are manipulated (deepfakes). While the multimedia forensics community has developed detectors to identify these fakes, most existing tools operate on a single modality (either audio or video) at a time. This monomodal approach is insufficient for detecting advanced deepfakes that exploit inconsistencies between audio and visual cues. A key bottleneck in developing multimodal detectors is the scarcity of datasets containing forgeries where both audio and video are manipulated. Existing datasets often focus on only one falsified modality, leaving a critical gap for training and testing robust multimodal forensic systems.

Methods

The core contribution is a pipeline for generating multimodal deepfake audio. This pipeline takes a video (V) as input, which consists of an audio track (x) and a visual component (y).

  1. Speech-to-Text: The original audio track (x) is transcribed into text (T).
  2. Text-to-Speech (TTS) Synthesis: A chosen TTS system generates a synthetic audio track (s) from the text (T). The paper utilizes 12 state-of-the-art TTS models, including popular ones like Tacotron, Tacotron2, GlowTTS, FastSpeech2, and VITS, to ensure a diverse range of synthetic speech characteristics.
  3. Post-processing: To enhance realism and challenge detectors, two post-processing steps are applied:
  4. Audio-Video Synchronization (DTW): Dynamic Time Warping is used to align the synthetic audio track (s) with the original audio track (x) to ensure temporal coherence with the video. This is crucial because asynchronous audio and video are easily detectable.
  5. Data Augmentation: Techniques like Gaussian noise injection, pitch shifting, and MP3 compression are applied to further obscure synthetic artifacts and simulate real-world degradation.

The resulting synthetic audio track (î) is then combined with the original video component (y) to create a multimodal deepfake.

Datasets

  • VidTIMIT: This dataset serves as the reference for real audio-video recordings. It contains 430 videos of 43 individuals speaking short sentences. The authors chose VidTIMIT because it provides official transcripts, eliminating the need for error-prone automatic transcription.
  • DeepfakeTIMIT: This dataset, previously released, contains video deepfakes derived from VidTIMIT.
  • TIMIT-TTS (New Dataset): This is the primary contribution of the paper. It's a novel multimodal deepfake dataset generated by applying the described pipeline to VidTIMIT and DeepfakeTIMIT. It comprises approximately 80,000 synthetic audio tracks, created using 12 different TTS generators, and is released in four versions: clean (no post-processing), DTW-synchronized, augmented, and DTW-synchronized + augmented.

Key Insights and Ideas

  • The Necessity of Multimodal Data: The experiments clearly demonstrate that monomodal detectors struggle with post-processed deepfakes, and multimodal analysis offers a significant advantage.
  • DTW's Impact on Detectability: While DTW synchronization makes audio more aligned with video, it paradoxically makes the synthetic audio harder to detect by audio-only detectors, highlighting the need for detectors trained on such synchronized data.
  • Speaker Identity as a Distractor: When TTS models are trained on a single speaker (like LJSpeech), detectors tend to rely on speaker identity rather than synthetic artifacts for classification. This suggests that datasets with diverse speakers across multiple TTS models are crucial for developing robust attribution methods that focus on generator-specific traces.
  • Real-world Data is Messy: The significant drop in detection performance when dealing with compressed video (QP=40) underscores the importance of developing detectors robust to real-world degradation.

Key Results

  • Audio Binary Classification: The AUC for detecting deepfakes dropped from 0.98 (clean) to 0.77 (DTW + augmented), indicating that post-processing significantly challenges monomodal audio detectors.
  • Audio Attribution: Identifying the specific TTS generator used for synthesis was more challenging for models trained on a single speaker (LJSpeech) compared to multi-speaker models. Balanced accuracy dropped from 0.77 to 0.67 for single-speaker models.
  • Video Binary Classification: Detection performance also degraded with compression, with AUC dropping from 0.99 (high quality) to 0.79 (low quality).
  • Multimodal Classification: Multimodal approaches consistently outperformed monomodal ones. In Scenario 2 (mixed real/fake modalities), multimodal detection achieved an AUC improvement of 0.15 over single modalities, even with simple score fusion.

Limitations and Future Work

  • Limited TTS Models: The dataset currently includes 12 TTS models. Future work could incorporate more models and explore the impact of different vocoders.
  • Limited Speaker Diversity: While TIMIT-TTS offers multiple speakers, the authors acknowledge that some TTS models are limited to the LJSpeech speaker. Future versions could aim for more speakers across all models.
  • Exclusion of Voice Conversion (VC): The current pipeline focuses on TTS. Future work plans to include VC methods to generate a more comprehensive dataset.
  • Simple Fusion Methods: The multimodal fusion used in experiments was basic (averaging or taking the max score). More sophisticated fusion techniques could further improve detection performance.
  • Need for More Data: The authors emphasize that while TIMIT-TTS is a significant step, more multimodal deepfake datasets are needed to train and test robust detectors for in-the-wild conditions.

Summary updated

Abstract

With the rapid development of deep learning techniques, the generation and counterfeiting of multimedia material has become increasingly simple. Current technology enables the creation of videos where both the visual and audio contents are falsified. While the multimedia forensics community has begun to address this threat by developing fake media detectors. However, the vast majority existing forensic techniques only analyze one modality at a time. This is an important limitation when authenticating manipulated videos, because sophisticated forgeries may be difficult to detect without exploiting cross-modal inconsistencies (e.g., across the audio and visual tracks). One important reason for the lack of multimodal detectors is a similar lack of research datasets containing multimodal forgeries. Existing datasets typically contain only one falsified modality, such as deepfaked videos with authentic audio tracks, or synthetic audio with no associated video. Currently, datasets are needed that can be used to develop, train, and test these forensic algorithms. In this paper, we propose a new audio-visual deepfake dataset containing multimodal video forgeries. We present a general pipeline for synthesizing deepfake speech content from a given video, facilitating the creation of counterfeit multimodal material. The proposed method uses Text-to-Speech (TTS) and Dynamic Time Warping (DTW) techniques to achieve realistic speech tracks. We use this pipeline to generate and release TIMIT-TTS, a synthetic speech dataset containing the most cutting-edge methods in the TTS field. This can be used as a standalone audio dataset, or combined with DeepfakeTIMIT and VidTIMIT video datasets to perform multimodal research. Finally, we present numerous experiments to benchmark the proposed dataset in both monomodal (i.e., audio) and multimodal (i.e., audio and video) conditions. This highlights the need for multimodal forensic detectors and more multimodal deepfake data.
BibTeX
@inproceedings{salvi2023timit,
  author = {Davide Salvi and Brian C. Hosler and Paolo Bestagini and Matthew C. Stamm and Stefano Tubaro},
  title = {{TIMIT-TTS: A Text-to-Speech Dataset for Multimodal Synthetic Media Detection}},
  booktitle = {IEEE Access},
  year = {2023},
  url = {https://openreview.net/forum?id=Ft5LiWDkG9},
}
Figure