Abstract
Recent techniques for speech deepfake detection often rely on pre-trained self-supervised models. These systems, initially developed for Automatic Speech Recognition (ASR), have proved their ability to offer a meaningful representation of speech signals, which can benefit various tasks, including deepfake detection. In this context, pre-trained models serve as feature extractors and are used to extract embeddings from input speech, which are then fed to a binary speech deepfake detector. The remarkable accuracy achieved through this approach underscores a potential relationship between ASR and speech deepfake detection. However, this connection is not yet entirely clear, and we do not know whether improved performance in ASR corresponds to higher speech deepfake detection capabilities. In this paper, we address this question through a systematic analysis. We consider two different pre-trained self-supervised ASR models, Whisper and Wav2Vec 2.0, and adapt them for the speech deepfake detection task. These models have been released in multiple versions, with increasing number of parameters and enhanced ASR performance. We investigate whether performance improvements in ASR correlate with improvements in speech deepfake detection. Our results provide insights into the relationship between these two tasks and offer valuable guidance for the development of more effective speech deepfake detectors.
Introduction
As synthetic speech technologies—specifically Text-to-Speech (TTS) and Voice Conversion (VC)—reach a level of realism indistinguishable from human voices, the forensic community faces a critical challenge. To combat this, researchers have increasingly turned to powerful, pre-trained self-supervised learning (SSL) models. These models, such as OpenAI’s Whisper and Meta’s Wav2Vec 2.0, are trained on massive datasets to master the nuances of human speech for Automatic Speech Recognition (ASR).
A prevailing assumption in the field is that a model better at "understanding" speech (measured by a lower Word Error Rate, or WER) will inherently be better at extracting the subtle acoustic features necessary to identify a deepfake. However, this hypothesized link between ASR accuracy and forensic capability has lacked rigorous testing. This study investigates whether there is a direct correlation between ASR performance and deepfake detection, or if the community is approaching a point of diminishing returns regarding model scaling.
Methodology
The research employs a systematic analysis by adapting two prominent ASR architectures: Whisper and Wav2Vec 2.0.
Architecture and Training
The authors utilized these ASR models as "frozen" feature extractors. In this setup, raw audio is fed into the pre-trained backbones to generate high-dimensional embeddings. These embeddings are then passed to a simple, trainable classification head consisting of fully connected layers to produce a binary "Real vs. Fake" decision. By keeping the ASR backbones frozen, the researchers ensured that any variation in detection performance was strictly attributable to the quality of the representations learned during the original ASR training phase.
Model Variants
To test the impact of scaling, multiple versions of these models were evaluated:
- Whisper: Ranging from "Tiny" (39M parameters) to "Large" (1550M parameters).
- Wav2Vec 2.0: Including various sizes and the multilingual
xls-rvariant.
Experimental Setup
The study utilized a comprehensive suite of benchmarks to evaluate both in-distribution performance and "in-the-wild" generalization:
- ASVspoof 2019 (Logical Access): The primary dataset for training and initial benchmarking.
- ASVspoof 2021 (DeepFake): Used to test robustness against lossy codecs common in social media environments.
- AISEC "In-the-Wild": Real-world clips featuring celebrities and politicians.
- TIMIT-TTS & LJspeech: Used to evaluate the ability to distinguish specific real speakers from their synthetic counterparts.
- FakeOrReal: A dataset containing a mix of open-source and commercial TTS systems.
Key Insights and Results
The Saturation Point
The most significant finding is that forensic performance does not scale infinitely with model size. For the Whisper architecture, the "Small" model (244M parameters) achieved the best average performance across all datasets. Interestingly, the "Large" model, despite having six times more parameters and a 20% better WER, often exhibited inferior performance. This suggests that the features required for high-level linguistic understanding may eventually diverge from the low-level acoustic artifacts necessary for forensic detection.
The ASR-Forensics Paradox
While a lower WER generally correlates with better detection in smaller models, the relationship breaks down at the higher end of the scale. As models become more optimized for what is being said (content), they may inadvertently begin to ignore the subtle hardware or algorithmic artifacts that reveal how the audio was generated (forensics).
Detection Diversity and Non-Hierarchical Behavior
The study revealed a surprising lack of detection overlap between different model sizes. One might expect a large model to subsume the capabilities of a smaller one; however, 75.6% of the tracks missed by Whisper Large were correctly identified by Whisper Tiny. This indicates that different model scales focus on entirely different types of audio artifacts.
The Multilingual Penalty
In Wav2Vec 2.0 experiments, the xls-r model (trained on 128 languages) performed significantly worse on English deepfake detection than the English-only "Large" version, despite sharing the same architecture size. This suggests that broad linguistic generalization may dilute the model's ability to capture specific forensic signatures within a single language.
Conclusion
The study concludes that Whisper-based detectors are highly effective, with Area Under Curve (AUC) values consistently above 98% on standard benchmarks. However, the Whisper "Small" model emerged as the optimal choice, yielding the lowest average Equal Error Rate (EER) of 13.23% across all test sets.
Future work should explore whether "unfreezing" (fine-tuning) larger models can overcome the observed performance plateau. Additionally, the results suggest a need for new feature engineering strategies that exploit speech-related information specifically tuned for forensic artifacts rather than linguistic content alone.
Summary updated