Abstract
End-to-end speaker verification (SV) systems are increasingly used in biometric security but remain vulnerable to adversarial attacks—imperceptible perturbations that can cause misidentification. This paper proposes Adversarial Regularization, incorporating adversarial examples generated via Fast Gradient-Sign Method (FGSM) and Local Distributional Smoothness (LDS) into the training process. Results on the TIMIT corpus demonstrate that this approach not only enhances robustness against attacks but also improves accuracy on clean data, achieving up to an 18.89% relative reduction in Equal Error Rate (EER).
Introduction
Speaker Verification (SV) confirms a person's identity based on their voice. While deep learning models, particularly those using Generalized End-to-End (GE2E) loss, have achieved high performance, they are susceptible to adversarial examples. Unlike traditional spoofing, these attacks involve adding mathematically calculated noise that is inaudible to humans but catastrophic for neural networks. This work aims to bridge this security gap by smoothing model decision boundaries through adversarial regularization.
Methodology
The core architecture utilizes a 3-layer LSTM network trained with GE2E loss. To improve robustness, the authors introduce two regularization techniques:
- Fast Gradient-Sign Method (FGSM): A supervised method that perturbs inputs in the direction of maximum loss increase.
- Local Distributional Smoothness (LDS): An unsupervised method (Virtual Adversarial Training) that minimizes the KL-divergence between the model's output for original and perturbed inputs.
The training process alternates between standard GE2E loss and adversarial loss, forcing the network to learn stable features that are resilient to slight input manipulations.
Experimental Setup
The study uses the TIMIT corpus (630 speakers) to isolate the impact of adversarial attacks from environmental noise. The data is split into 540 speakers for training, 30 for validation, and 60 for testing. By using a phonetically balanced dataset without significant background noise, the authors could precisely quantify the vulnerability of the SV system.
Results and Discussion
Adversarial regularization provides a "double win": it increases robustness to attacks and improves performance on original data. The baseline model's EER jumped from 4.87% to 11.89% under attack, whereas regularized models maintained significantly lower error rates.
Key findings include:
- FGSM-REG achieved the best performance on clean data with a 3.95% EER (18.89% relative improvement).
- LDS-REG improved clean data performance to 4.60% EER.
- Robustness to FGSM attacks improved by up to 30.11% relative to the baseline, reducing the EER from 11.89% to 8.31%.
Conclusion
By training on the neighborhood of data points rather than isolated points, adversarial regularization creates more generalized speaker embeddings. This suggests that forcing the model to be robust to "worst-case" noise helps it learn higher-quality features. Future work will explore larger "in-the-wild" datasets and more complex iterative attacks.
Summary updated