Adversarial Speaker Verification

Zhong Meng, Yong Zhao, Jinyu Li, Yifan Gong

Abstract

Adversarial Speaker Verification (ASV) addresses the challenge of domain mismatch in speaker recognition by using adversarial multi-task training. By incorporating a Gradient Reversal Layer (GRL), the system forces the feature extractor to produce embeddings that are invariant to environmental noise and signal-to-noise ratios (SNR). Evaluation on large-scale real-world datasets shows that ASV significantly outperforms standard data augmentation techniques, especially in unseen acoustic environments.

Introduction

Deep learning has revolutionized speaker verification (SV) through the use of "embeddings"—compact vector representations of a person's voice. However, these embeddings often suffer from domain mismatch. If a system is trained on clean speech but deployed in a noisy environment, the speaker signal becomes entangled with the acoustic condition. While data augmentation helps, models still tend to learn features specific to certain noise profiles. The challenge is to extract a "canonical" speaker representation that is truly condition-invariant, meaning the embedding for a specific person remains consistent regardless of the environment.

Methodology

The authors propose a multi-task DNN architecture consisting of three main components: a Feature Extractor ($M_f$), a Speaker Classifier ($M_y$), and a Condition Network ($M_c$).

The adversarial mechanism is implemented via a Gradient Reversal Layer (GRL) placed between the feature extractor and the condition network. During the forward pass, the GRL acts as an identity transform. During the backward pass, it multiplies the gradient by a negative scalar ($-\lambda$). This setup creates a minimax game: the condition network tries to improve its identification of the noise or SNR, while the feature extractor is pushed to modify its representations so that the condition network fails. Effectively, the feature extractor is trained to "hide" environmental information while retaining speaker-discriminative features.

Figure 1: Architecture of the Adversarial Speaker Verification (ASV) system featuring the Gradient Reversal Layer (GRL).

The framework explores different condition networks:

  • Categorical: A classification head for discrete environment types (e.g., Bus, Cafe).
  • Continuous: A regression head for continuous values like SNR levels.
  • Multi-factorial: A combined version using both categorical and continuous heads to suppress multiple sources of variability simultaneously.

Experimental Results

The researchers utilized a massive internal dataset from Microsoft Cortana, comprising 6.8 million utterances augmented to 20.4 million noisy samples. The system was compared against a strong baseline (DE) trained with heavy data augmentation.

In known conditions (Test A), ASV achieved an 8.8% relative improvement in Equal Error Rate (EER). The most significant gains were observed in unknown conditions (Test B), which included real-world noise from TVs and background music, as well as varying recording distances. In these unseen environments, ASV achieved a 14.5% relative improvement, reducing the total EER from 13.02% to 11.13%.

Figure 2: Performance (EER %) on Test B (Unknown Conditions). EI refers to Environment-invariant and SI refers to SNR-invariant training.

The results demonstrate that combining both Environment-invariant (EI) and SNR-invariant (SI) training (EI+SI DE) provides the best performance, yielding a ~5.3% additional gain over using a single adversarial factor. Interestingly, the adversarial approach generalizes better to unknown conditions than to known ones, suggesting it learns a more fundamental, noise-robust way of representing speech.

Conclusion

Adversarial training is a powerful method for achieving domain generalization in speaker verification. By explicitly training the model to "forget" unwanted environmental factors, ASV produces embeddings that are more robust to real-world variability. While the system showed some limitations with competing speech and extreme reverberation, it represents a significant step toward truly condition-invariant speaker representations.

Summary updated

Abstract

The use of deep networks to extract embeddings for speaker recognition has proven successfully. However, such embeddings are susceptible to performance degradation due to the mismatches among the training, enrollment, and test conditions. In this work, we propose an adversarial speaker verification (ASV) scheme to learn the condition-invariant deep embedding via adversarial multi-task training. In ASV, a speaker classification network and a condition identification network are jointly optimized to minimize the speaker classification loss and simultaneously mini-maximize the condition loss. The target labels of the condition network can be categorical (environment types) and continuous (SNR values). We further propose multi-factorial ASV to simultaneously suppress multiple factors that constitute the condition variability. Evaluated on a Microsoft Cortana text-dependent speaker verification task, the ASV achieves 8.8% and 14.5% relative improvements in equal error rates (EER) for known and unknown conditions, respectively.
BibTeX
@inproceedings{meng2019adversarial,
  author = {Zhong Meng and Yong Zhao and Jinyu Li and Yifan Gong},
  title = {{Adversarial Speaker Verification}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  doi = {10.1109/icassp.2019.8682488},
}
Figure