A Comparative Study on Proactive and Passive Detection of Deepfake Speech

Chia-Hua Wu, Wanying Ge, Xin Wang, Junichi Yamagishi, Yu Tsao, Hsin‐Min Wang

Abstract

Solutions for defending against deepfake speech fall into two categories: proactive watermarking models and passive conventional deepfake detectors. While both address common threats, their differences in training, optimization, and evaluation prevent a unified protocol for joint evaluation and selecting the best solutions for different cases. This work proposes a framework to evaluate both model types in deepfake speech detection. To ensure fair comparison and minimize discrepancies, all models were trained and tested on common datasets, with performance evaluated using a shared metric. We also analyze their robustness against various adversarial attacks, showing that different models exhibit distinct vulnerabilities to different speech attribute distortions. Our training and evaluation code is available at Github.

Introduction

Defending against AI-generated deepfake speech generally falls into two distinct camps. Passive defense involves training classifiers (deepfake detectors) to spot artifacts in fake audio after the fact. Proactive defense involves embedding imperceptible, traceable watermarks into authentic audio before distribution, allowing recipients to verify its source.

Because these approaches have fundamentally different objectives, training methods, and output formats (binary real/fake scores vs. multi-bit message extraction), their respective research communities have remained siloed. Until now, there has been no standard protocol to fairly compare a passive detector against a proactive watermarking model. This makes it difficult for engineers to determine which solution is better suited for real-world deployment, especially when audio is subjected to compression, background noise, or adversarial manipulation.

Unified Evaluation Methodology

The authors propose a unified evaluation pipeline that bridges the gap between the two paradigms. The key innovation is a scoring conversion method: they take the multi-bit output of a watermarking model and mathematically map it to a continuous "real vs. fake" likelihood score. This allows for the calculation of the Equal Error Rate (EER)—the standard evaluation metric used for passive deepfake detectors.

The proposed unified evaluation framework for comparing passive and proactive deepfake detection models.

They evaluated four representative models:

  • Passive Models: AASIST (a state-of-the-art end-to-end detector) and SSL-AASIST (which upgrades AASIST with a pre-trained wav2vec 2.0 front-end).
  • Proactive Models: Timbre (embeds watermarks in the spectral/frequency domain) and AudioSeal (a state-of-the-art sequence-to-sequence watermarking model).

To test robustness, they subjected the audio to two groups of distortions: noisy/public transmission channels (Gaussian noise, room impulse responses, and codecs like Opus and DAC) and acoustic manipulations (clipping, pitch shifting, time stretching, and overdrive).

Experimental Results

Experiments were conducted using the ASVspoof 2019 Logical Access (LA) dataset for training and validation, and the ASVspoof 2021 LA dataset to evaluate generalization under simulated lossy transmission channels.

Detailed EER (%) results across various transmission and manipulation conditions for both passive and proactive models.

Key Findings

  • Clean Audio Performance: Without background noise or manipulation, both passive (SSL-AASIST) and proactive (Timbre, AudioSeal) models achieve near-perfect detection. On the clean ASVspoof 2019 LA test set, proactive models achieved a 0.00% EER, while SSL-AASIST reached 0.23%.
  • Architecture-Specific Vulnerabilities: Passive models are highly vulnerable to temporal modifications like time-stretching (AASIST hit 66.53% EER). Conversely, Timbre (proactive) is robust to temporal changes due to its frequency-domain embedding but fails catastrophically under pitch shifting (52.62% EER).
  • Neural Codec Impact: Modern discrete neural audio codecs (like DAC or WavTokenizer) act as "deepfake camouflage." Passing audio through these codecs severely degrades performance across almost all models. For instance, AudioSeal's EER skyrocketed to 97.40% under DAC compression.
  • Robustness Winner: Timbre proved to be the most robust model overall, maintaining an average EER of 8.87% across all 2019 LA manipulation conditions.

Conclusion

While clean audio detection is largely a solved problem, surviving real-world distortions remains a challenge. Different architectures exhibit distinct "Achilles' heels," with neural codecs and pitch-shifting posing the greatest threats to current state-of-the-art systems. Future work will investigate how uniform data augmentation strategies during training might improve cross-paradigm robustness.

Summary updated

BibTeX
@misc{wu2025comparative,
  author = {Chia-Hua Wu and Wanying Ge and Xin Wang and Junichi Yamagishi and Yu Tsao and Hsin‐Min Wang},
  title = {{A Comparative Study on Proactive and Passive Detection of Deepfake Speech}},
  year = {2025},
  eprint = {2506.14398},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.14398},
}
Figure