Abstract
Detecting synthetic from real speech is increasingly crucial due to the risks of misinformation and identity impersonation. While various datasets for synthetic speech analysis have been developed, they often focus on specific areas, limiting their utility for comprehensive research. To fill this gap, we propose the Speech-Forensics dataset by extensively covering authentic, synthetic, and partially forged speech samples that include multiple segments synthesized by different high-quality algorithms. Moreover, we propose a TEmporal Speech LocalizaTion network, called TEST, aiming at simultaneously performing authenticity detection, multiple fake segments localization, and synthesis algorithms recognition, without any complex post-processing. TEST effectively integrates LSTM and Transformer to extract more powerful temporal speech representations and utilizes dense prediction on multi-scale pyramid features to estimate the synthetic spans. Our model achieves an average mAP of 83.55% and an EER of 5.25% at the utterance level. At the segment level, it attains an EER of 1.07% and a 92.19% F1 score. These results highlight the model's robust capability for a comprehensive analysis of synthetic speech, offering a promising avenue for future research and practical applications in this field.
Introduction
The rapid evolution of text-to-speech (TTS) and voice conversion (VC) technologies has created a "cat and mouse" game in the field of speech forensics. Modern deepfake audio is now nearly indistinguishable from human speech to the naked ear. However, most contemporary detection tools are designed for a binary paradigm: determining if an entire clip is real or fake.
In real-world forensic scenarios, malicious actors rarely synthesize entire recordings. Instead, they often employ "partial forgery," where specific words or phrases are swapped within an otherwise authentic recording to alter its meaning. This paper addresses the limitations of current datasets and models by introducing the Speech-Forensics dataset and the TEST (TEmporal Speech LocalizaTion) network, designed to detect, localize, and attribute these sophisticated manipulations.
The Speech-Forensics Dataset
Existing datasets often suffer from over-simplification, focusing on fully synthetic clips or single-segment modifications. To create a more realistic challenge, the authors developed the Speech-Forensics dataset based on the LJ Speech corpus.
Construction Strategy
The dataset utilizes a "keyword replacement" strategy to ensure semantic coherence. Using Named Entity Recognition (NER), the authors identified names and organizations, and used an antonym strategy for adjectives. These specific segments were then replaced with synthetic versions generated by 11 different high-quality algorithms, including Tacotron2, Glow-TTS, and VITS.
Unlike previous benchmarks, a single clip in the Speech-Forensics dataset can contain multiple forged segments, each potentially generated by a different synthesis model. This multi-segment, multi-algorithm approach mirrors the complexity of actual forensic evidence.
The TEST Network Architecture
To analyze this complex data, the authors propose TEST, a multi-task framework inspired by temporal action localization in computer vision. The architecture is designed to perform detection, localization, and algorithm recognition in a single pass.
Key Components
- Feature Embedding: The model supports handcrafted features (MFCC/LFCC) or high-level representations from Self-Supervised Learning (SSL) models such as Wav2Vec2 or WavLM.
- Masked Difference Convolution: This layer emulates first- and second-order derivatives to enrich temporal information before processing.
- R-Transformer Backbone: By combining LSTM (for sequential dependencies) and Transformers (for global context), the backbone extracts powerful temporal representations.
- Feature Pyramid Network (FPN): To detect forged segments of varying lengths—from a single syllable to a full sentence—the model utilizes a multi-scale pyramid of features.
- Dense Prediction: TEST employs an "anchor-free" regression method to estimate the start and end times of forged spans directly, avoiding the noise associated with frame-by-frame classification.
Experimental Results and Insights
The TEST model demonstrates significant improvements over traditional baselines like RawNet2 and AASIST. On the Speech-Forensics dataset, the model achieved:
- Utterance-level EER: 5.25%
- Segment-level EER: 1.07%
- Segment-level F1 Score: 92.19%
- Localization mAP: 83.55% across various Intersection over Union (IoU) thresholds.
The "Shallow Feature" Advantage
A critical insight from the research is the performance of SSL features. The authors found that features from the shallower layers of models like WavLM yielded superior results. While deeper layers capture semantic meaning (the "what"), shallower layers are more sensitive to the acoustic artifacts and glitches (the "how") left behind by synthesis algorithms. When using WavLM-Large as a feature extractor, the utterance-level EER dropped to an impressive 0.64%.
Conclusion
The Speech-Forensics dataset and the TEST network represent a significant advancement in audio deepfake detection. By moving beyond binary classification toward precise temporal localization and algorithm attribution, this research provides a more robust toolkit for forensic experts.
Future work will focus on addressing current limitations, such as expanding speaker diversity beyond the single-speaker LJ Speech base and improving model robustness against environmental factors like background noise and telephone-grade compression, which are common in real-world forensic investigations.
Summary updated