Abstract
Solutions for defending against deepfake speech fall into two categories: proactive watermarking models and passive conventional deepfake detectors. While both address common threats, their differences in training, optimization, and evaluation prevent a unified protocol for joint evaluation and selecting the best solutions for different cases. This work proposes a framework to evaluate both model types in deepfake speech detection. To ensure fair comparison and minimize discrepancies, all models were trained and tested on common datasets, with performance evaluated using a shared metric. We also analyze their robustness against various adversarial attacks, showing that different models exhibit distinct vulnerabilities to different speech attribute distortions. Our training and evaluation code is available at Github.
Introduction
Defending against AI-generated deepfake speech generally falls into two distinct camps. Passive defense involves training classifiers (deepfake detectors) to spot artifacts in fake audio after the fact. Proactive defense involves embedding imperceptible, traceable watermarks into authentic audio before distribution, allowing recipients to verify its source.
Because these approaches have fundamentally different objectives, training methods, and output formats (binary real/fake scores vs. multi-bit message extraction), their respective research communities have remained siloed. Until now, there has been no standard protocol to fairly compare a passive detector against a proactive watermarking model. This makes it difficult for engineers to determine which solution is better suited for real-world deployment, especially when audio is subjected to compression, background noise, or adversarial manipulation.
Unified Evaluation Methodology
The authors propose a unified evaluation pipeline that bridges the gap between the two paradigms. The key innovation is a scoring conversion method: they take the multi-bit output of a watermarking model and mathematically map it to a continuous "real vs. fake" likelihood score. This allows for the calculation of the Equal Error Rate (EER)—the standard evaluation metric used for passive deepfake detectors.
They evaluated four representative models:
- Passive Models: AASIST (a state-of-the-art end-to-end detector) and SSL-AASIST (which upgrades AASIST with a pre-trained wav2vec 2.0 front-end).
- Proactive Models: Timbre (embeds watermarks in the spectral/frequency domain) and AudioSeal (a state-of-the-art sequence-to-sequence watermarking model).
To test robustness, they subjected the audio to two groups of distortions: noisy/public transmission channels (Gaussian noise, room impulse responses, and codecs like Opus and DAC) and acoustic manipulations (clipping, pitch shifting, time stretching, and overdrive).
Experimental Results
Experiments were conducted using the ASVspoof 2019 Logical Access (LA) dataset for training and validation, and the ASVspoof 2021 LA dataset to evaluate generalization under simulated lossy transmission channels.
Key Findings
- Clean Audio Performance: Without background noise or manipulation, both passive (SSL-AASIST) and proactive (Timbre, AudioSeal) models achieve near-perfect detection. On the clean ASVspoof 2019 LA test set, proactive models achieved a 0.00% EER, while SSL-AASIST reached 0.23%.
- Architecture-Specific Vulnerabilities: Passive models are highly vulnerable to temporal modifications like time-stretching (AASIST hit 66.53% EER). Conversely, Timbre (proactive) is robust to temporal changes due to its frequency-domain embedding but fails catastrophically under pitch shifting (52.62% EER).
- Neural Codec Impact: Modern discrete neural audio codecs (like DAC or WavTokenizer) act as "deepfake camouflage." Passing audio through these codecs severely degrades performance across almost all models. For instance, AudioSeal's EER skyrocketed to 97.40% under DAC compression.
- Robustness Winner: Timbre proved to be the most robust model overall, maintaining an average EER of 8.87% across all 2019 LA manipulation conditions.
Conclusion
While clean audio detection is largely a solved problem, surviving real-world distortions remains a challenge. Different architectures exhibit distinct "Achilles' heels," with neural codecs and pitch-shifting posing the greatest threats to current state-of-the-art systems. Future work will investigate how uniform data augmentation strategies during training might improve cross-paradigm robustness.
Summary updated