ASVspoof 2019: spoofing countermeasures for the detection of synthesized, converted and replayed speech

Andreas Nautsch, Xin Wang, Nicholas W. D. Evans, Tomi Kinnunen, Ville Vestman, Massimiliano Todisco, Héctor Delgado, Md. Sahidullah, Junichi Yamagishi, Kong Aik Lee

Abstract

The ASVspoof initiative was conceived to spearhead research in anti-spoofing for automatic speaker verification (ASV). This paper describes the third in a series of bi-annual challenges: ASVspoof 2019. The focus of this paper is on results and the top performing single and ensemble system submissions from 62 teams, all of which out-perform the two baseline systems. Deeper analyses shows that performance is dominated by specific conditions involving either specific spoofing attacks or specific acoustic environments. While fusion is effective for logical access scenarios, participants struggled to apply fusion successfully for physical access scenarios involving simulated replay attacks. Experiments with real replay data show a substantial performance gap compared to simulated data, likely due to additive noise.

Introduction

Automatic Speaker Verification (ASV) systems—the technology enabling voice-based authentication—are vulnerable to "presentation attacks" (spoofing). Impostors may use text-to-speech (TTS), voice conversion (VC), or simple recordings (replay) to gain unauthorized access.

The ASVspoof 2019 challenge addresses three critical gaps:

  1. The "Tandem" Problem: Moving beyond isolated countermeasure evaluation to the t-DCF (tandem Detection Cost Function), which assesses how well a countermeasure protects a specific ASV system.
  2. Attack Diversity: Covering both Logical Access (LA) (digitally injected synthetic speech) and Physical Access (PA) (replaying recordings into a microphone).
  3. The Simulation Gap: Investigating whether models trained on simulated replay data generalize to noisy, real-world acoustic environments.

Methodology

The challenge saw a definitive shift from classical signal processing toward deep learning. Top-performing systems utilized Light Convolutional Neural Networks (LCNN), ResNets, and Squeeze-and-Excitation networks (SENet).

Feature Engineering and Fusion

While standard features like LFCC (Linear Frequency Cepstral Coefficients) remained popular, winners often used "spectrogram-like" inputs (CQT-grams, Mel-spectrograms) or Group Delay grams to capture phase inconsistencies. The most successful approach was ensemble modeling, fusing multiple sub-systems to catch different types of artifacts.

Datasets and Attack Scenarios

The challenge used the ASVspoof 2019 database, sourced from the VCTK corpus.

Logical Access (LA)

The LA set included 17 different TTS and VC systems, ranging from older waveform concatenation to state-of-the-art neural vocoders like WaveNet.

Comparison of Text-to-Speech (TTS) and Voice Conversion (VC) pipelines in the Logical Access scenario.

Physical Access (PA)

The PA set used a simulation pipeline modeling room acoustics (image-source method) and replay hardware effects (Hammerstein model). A "Hidden Real Replay" subset contained actual recordings made in physical rooms to test "sim-to-real" generalization.

Simulated vs. Real-world replay attack configurations for Physical Access.

Key Results and Insights

The results highlighted a significant disparity between synthetic and replay detection performance.

  • Logical Access (LA): The top system achieved a min t-DCF of 0.0692 and an Equal Error Rate (EER) of just 0.22%. Fusion was highly effective here because different models caught distinct "digital signatures" of various synthesis engines.
  • Physical Access (PA): Performance was lower, with the best system reaching a min t-DCF of 0.1437 and an EER of 0.39%. Fusion provided diminishing returns, suggesting current features are redundant in identifying acoustic artifacts.

The "Sim-to-Real" Wall

Systems that performed flawlessly on simulated replay data struggled on real-world recordings, where the best t-DCF jumped to 0.3855. The primary culprit is additive noise (hums, fans, ambient sounds) which masks the subtle artifacts models rely on. Furthermore, replay detection is often dominated by the "room signature," as the signal propagates through the environment twice.

Future Directions

The authors identify several frontiers for the next generation of anti-spoofing:

  • Robustness to Noise: Challenges must include "in-the-wild" conditions with varying signal-to-noise ratios.
  • Explainability: Moving beyond "black box" neural networks to understand what artifacts (e.g., neural vocoder glitches) the models are actually detecting.
  • New Attack Vectors: Addressing adversarial attacks and "blended" attacks (replaying synthetic voices).

Summary updated

Abstract

The ASVspoof initiative was conceived to spearhead research in anti-spoofing for automatic speaker verification (ASV). This paper describes the third in a series of bi-annual challenges: ASVspoof 2019. With the challenge database and protocols being described elsewhere, the focus of this paper is on results and the top performing single and ensemble system submissions from 62 teams, all of which out-perform the two baseline systems, often by a substantial margin. Deeper analyses shows that performance is dominated by specific conditions involving either specific spoofing attacks or specific acoustic environments. While fusion is shown to be particularly effective for the logical access scenario involving speech synthesis and voice conversion attacks, participants largely struggled to apply fusion successfully for the physical access scenario involving simulated replay attacks. This is likely the result of a lack of system complementarity, while oracle fusion experiments show clear potential to improve performance. Furthermore, while results for simulated data are promising, experiments with real replay data show a substantial gap, most likely due to the presence of additive noise in the latter. This finding, among others, leads to a number of ideas for further research and directions for future editions of the ASVspoof challenge.
BibTeX
@article{nautsch2021asvspoof,
  author = {Andreas Nautsch and Xin Wang and Nicholas W. D. Evans and Tomi Kinnunen and Ville Vestman and Massimiliano Todisco and Héctor Delgado and Md. Sahidullah and Junichi Yamagishi and Kong Aik Lee},
  title = {{ASVspoof 2019: spoofing countermeasures for the detection of synthesized, converted and replayed speech}},
  journal = {IEEE Trans. Biom. Behav. Identity Sci.},
  year = {2021},
  url = {https://openreview.net/forum?id=lAWlf8BbZS},
}
Figure