Speech-Forensics: Towards Comprehensive Synthetic Speech Dataset Establishment and Analysis

Zhoulin Ji, Chenhao Lin, Hang Wang, Chao Shen

Abstract

Detecting synthetic from real speech is increasingly crucial due to the risks of misinformation and identity impersonation. While various datasets for synthetic speech analysis have been developed, they often focus on specific areas, limiting their utility for comprehensive research. To fill this gap, we propose the Speech-Forensics dataset by extensively covering authentic, synthetic, and partially forged speech samples that include multiple segments synthesized by different high-quality algorithms. Moreover, we propose a TEmporal Speech LocalizaTion network, called TEST, aiming at simultaneously performing authenticity detection, multiple fake segments localization, and synthesis algorithms recognition, without any complex post-processing. TEST effectively integrates LSTM and Transformer to extract more powerful temporal speech representations and utilizes dense prediction on multi-scale pyramid features to estimate the synthetic spans. Our model achieves an average mAP of 83.55% and an EER of 5.25% at the utterance level. At the segment level, it attains an EER of 1.07% and a 92.19% F1 score. These results highlight the model's robust capability for a comprehensive analysis of synthetic speech, offering a promising avenue for future research and practical applications in this field.

Introduction

As deepfake audio becomes indistinguishable from human speech, forensic experts face a "cat-and-mouse" game. Most existing research focuses on utterance-level detection (determining if an entire clip is real or fake). However, real-world misinformation often involves partial forgeries, where a legitimate recording is tampered with by replacing specific words or phrases to change the meaning (e.g., changing "I did not" to "I did").

Existing datasets often oversimplify these scenarios by only including a single forged region or failing to provide labels for the specific algorithms used. This research addresses the need for a multi-dimensional forensic analysis: identifying not just if a clip is forged, but where the edits are and what tools were used to create them.

The TEST Architecture

The proposed TEmporal Speech LocalizaTion (TEST) network is a unified neural network designed to handle detection, localization, and recognition simultaneously. Its design is inspired by temporal action localization in computer vision.

Embedding and Backbone

TEST uses a mix of handcrafted features (LFCC/MFCC) and deep features from self-supervised learning (SSL) models like Wav2Vec2 and WavLM. A unique projection layer uses masked difference convolutions to capture temporal gradients, mimicking traditional forensic techniques that look for inconsistencies between frames. The core backbone consists of R-Transformer blocks that integrate LSTMs into the Transformer architecture, allowing the model to capture both long-range dependencies and local temporal patterns.

Multi-Scale Detection

The model constructs a multi-scale feature pyramid to detect forged segments of varying lengths—from short words to long sentences. Unlike frame-by-frame classifiers, TEST performs "anchor-free" dense regression. It predicts the distance from any given timestamp to the start and end of a forged span, allowing it to handle multiple overlapping or adjacent forgeries gracefully.

The Speech-Forensics Dataset

The Speech-Forensics dataset was built upon the LJ Speech corpus to provide a more realistic benchmark for partial forgery detection. To ensure semantic coherence, the authors used Named Entity Recognition (NER) and antonym replacement to choose which words to forge. These words were then synthesized using 11 different state-of-the-art algorithms (including Tacotron2, Glow-TTS, and VITS) and spliced back into the original audio.

Statistical distribution of the Speech-Forensics dataset, showing (a) duration time, (b) span length, and (c) algorithm count per sample.

As shown in the figure above, the dataset contains a diverse range of clip durations and forged span lengths. Notably, many samples include multiple forged segments created by different algorithms, reflecting the complexity of modern audio tampering.

Key Insights and Results

A significant finding of this study is the "Semantic Trap": shallower layers of SSL models (like Wav2Vec2) often perform better for forgery detection than deeper layers. This is because deep layers focus on high-level semantic meaning, while forgeries are often best detected by looking at low-level acoustic artifacts and boundary inconsistencies that deeper layers might ignore.

The TEST model demonstrated high precision across multiple metrics:

  • Localization & Recognition: Achieved a Mean Average Precision (mAP) of 83.55%.
  • Utterance-level Detection: An Equal Error Rate (EER) of 5.25%.
  • Segment-level Accuracy: At a granular level (0.01s segments), it achieved a remarkably low EER of 1.07% and an F1 score of 92.19%.
  • Feature Power: Using WavLM-Large as a feature extractor yielded the best results, pushing the utterance-level EER down to 0.64%.

Standard anti-spoofing models like RawNet2 and AASIST, designed for fully synthetic speech, struggle significantly (EERs over 24%) when faced with the partially forged samples in this new dataset, highlighting the necessity of the TEST approach.

Conclusion

Speech-Forensics and the TEST model represent a step forward in comprehensive synthetic speech analysis. By treating detection, localization, and recognition as a unified task, the model learns a more robust representation of forgery. Future work will focus on expanding speaker diversity and improving generalization to unseen synthesis algorithms.

Summary updated

Abstract

Detecting synthetic from real speech is increasingly crucial due to the risks of misinformation and identity impersonation. While various datasets for synthetic speech analysis have been developed, they often focus on specific areas, limiting their utility for comprehensive research. To fill this gap, we propose the Speech-Forensics dataset by extensively covering authentic, synthetic, and partially forged speech samples that include multiple segments synthesized by different high-quality algorithms. Moreover, we propose a TEmporal Speech LocalizaTion network, called TEST, aiming at simultaneously performing authenticity detection, multiple fake segments localization, and synthesis algorithms recognition, without any complex post-processing. TEST effectively integrates LSTM and Transformer to extract more powerful temporal speech representations and utilizes dense prediction on multi-scale pyramid features to estimate the synthetic spans. Our model achieves an average mAP of 83.55% and an EER of 5.25% at the utterance level. At the segment level, it attains an EER of 1.07% and a 92.19% F1 score. These results highlight the model's robust capability for a comprehensive analysis of synthetic speech, offering a promising avenue for future research and practical applications in this field.
BibTeX
@inproceedings{ji2024speech,
  author = {Zhoulin Ji and Chenhao Lin and Hang Wang and Chao Shen},
  title = {{Speech-Forensics: Towards Comprehensive Synthetic Speech Dataset Establishment and Analysis}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=e1gpUCtil0},
}
Figure