Abstract
Adversarial Speaker Verification (ASV) addresses the challenge of domain mismatch in speaker recognition by using adversarial multi-task training. By incorporating a Gradient Reversal Layer (GRL), the system forces the feature extractor to produce embeddings that are invariant to environmental noise and signal-to-noise ratios (SNR). Evaluation on large-scale real-world datasets shows that ASV significantly outperforms standard data augmentation techniques, especially in unseen acoustic environments.
Introduction
Deep learning has revolutionized speaker verification (SV) through the use of "embeddings"—compact vector representations of a person's voice. However, these embeddings often suffer from domain mismatch. If a system is trained on clean speech but deployed in a noisy environment, the speaker signal becomes entangled with the acoustic condition. While data augmentation helps, models still tend to learn features specific to certain noise profiles. The challenge is to extract a "canonical" speaker representation that is truly condition-invariant, meaning the embedding for a specific person remains consistent regardless of the environment.
Methodology
The authors propose a multi-task DNN architecture consisting of three main components: a Feature Extractor ($M_f$), a Speaker Classifier ($M_y$), and a Condition Network ($M_c$).
The adversarial mechanism is implemented via a Gradient Reversal Layer (GRL) placed between the feature extractor and the condition network. During the forward pass, the GRL acts as an identity transform. During the backward pass, it multiplies the gradient by a negative scalar ($-\lambda$). This setup creates a minimax game: the condition network tries to improve its identification of the noise or SNR, while the feature extractor is pushed to modify its representations so that the condition network fails. Effectively, the feature extractor is trained to "hide" environmental information while retaining speaker-discriminative features.
The framework explores different condition networks:
- Categorical: A classification head for discrete environment types (e.g., Bus, Cafe).
- Continuous: A regression head for continuous values like SNR levels.
- Multi-factorial: A combined version using both categorical and continuous heads to suppress multiple sources of variability simultaneously.
Experimental Results
The researchers utilized a massive internal dataset from Microsoft Cortana, comprising 6.8 million utterances augmented to 20.4 million noisy samples. The system was compared against a strong baseline (DE) trained with heavy data augmentation.
In known conditions (Test A), ASV achieved an 8.8% relative improvement in Equal Error Rate (EER). The most significant gains were observed in unknown conditions (Test B), which included real-world noise from TVs and background music, as well as varying recording distances. In these unseen environments, ASV achieved a 14.5% relative improvement, reducing the total EER from 13.02% to 11.13%.
The results demonstrate that combining both Environment-invariant (EI) and SNR-invariant (SI) training (EI+SI DE) provides the best performance, yielding a ~5.3% additional gain over using a single adversarial factor. Interestingly, the adversarial approach generalizes better to unknown conditions than to known ones, suggesting it learns a more fundamental, noise-robust way of representing speech.
Conclusion
Adversarial training is a powerful method for achieving domain generalization in speaker verification. By explicitly training the model to "forget" unwanted environmental factors, ASV produces embeddings that are more robust to real-world variability. While the system showed some limitations with competing speech and extreme reverberation, it represents a significant step toward truly condition-invariant speaker representations.
Summary updated