Abstract
In this paper, we propose a noise robust bottleneck feature representation which is generated by an adversarial network (AN). The AN includes two cascade connected networks, an encoding network (EN) and a discriminative network (DN). Mel-frequency cepstral coefficients (MFCCs) of clean and noisy speech are used as input to the EN and the output of the EN is used as the noise robust feature. The EN and DN are trained in turn, namely, when training the DN, noise types are selected as the training labels and when training the EN, all labels are set as the same, i.e., the clean speech label, which aims to make the AN features invariant to noise and thus achieve noise robustness. We evaluate the performance of the proposed feature on a Gaussian Mixture Model-Universal Background Model based speaker verification system, and make comparison to MFCC features of speech enhanced by short-time spectral amplitude minimum mean square error (STSA-MMSE) and deep neural network-based speech enhancement (DNN-SE) methods. Experimental results on the RSR2015 database show that the proposed AN bottleneck feature (AN-BN) dramatically outperforms the STSA-MMSE and DNN-SE based MFCCs for different noise types and signal-to-noise ratios. Furthermore, the AN-BN feature is able to improve the speaker verification performance under the clean condition.
Introduction
Speaker verification (SV) systems are increasingly deployed in real-world applications, yet their performance remains highly susceptible to environmental noise. Traditional approaches often utilize a front-end speech enhancement module, such as a Wiener filter or a deep neural network (DNN), to reconstruct clean audio before feature extraction. However, these methods are typically optimized for signal reconstruction rather than for the specific task of speaker discrimination. This work explores an alternative: learning a feature representation that is inherently invariant to noise through adversarial training.
Methodology: Adversarial Network Bottleneck (AN-BN)
The proposed system utilizes an Adversarial Network (AN) to extract noise-robust bottleneck features. The architecture, illustrated in Figure 1, consists of two primary components: an Encoding Network (EN) and a Discriminative Network (DN).
The EN takes 11 frames of MFCCs as input and produces a 128-dimensional bottleneck feature. The DN then attempts to classify the noise type (or clean speech) from this feature. The adversarial training process involves alternating steps:
- DN Training: The DN is trained to accurately identify noise types.
- EN Training: The EN is trained to map all inputs—regardless of noise level—to the "clean speech" label, effectively "fooling" the DN.
Through this competition, the EN learns to suppress noise-related information in the bottleneck features, creating a representation that is robust to acoustic interference.
Experimental Setup
The system was evaluated using the RSR2015 database (male speakers) with noises including White, Babble, Cantine, Market, and Airplane at SNRs from 0dB to 20dB. The AN-BN features were used as input for a standard Gaussian Mixture Model-Universal Background Model (GMM-UBM) SV system. Baselines included standard MFCCs, statistical enhancement (STSA-MMSE), and a DNN-based speech enhancement (DNN-SE) method.
Results and Discussion
Experimental results demonstrate that the AN-BN features significantly outperform traditional and deep learning-based enhancement methods. As shown in the results table below, the "Noise General" (NG-AN) model, trained on multiple noise types, consistently achieves the lowest Equal Error Rates (EER).
Key findings include:
- Superior Robustness: The AN-BN method reduced average EERs by approximately 25% to 40% compared to the DNN-SE baseline across various noise types.
- Generalization: Even though the AN was trained on moderate noise (10dB and 20dB), it maintained high performance on severe noise (0dB and 5dB).
- Clean Condition Improvement: Remarkably, the NG-AN features improved EER on clean speech to 1.33%, compared to 5.85% for the baseline MFCCs. This suggests the EN learns to remove session-specific variability in addition to environmental noise.
Conclusion
The Adversarial Network Bottleneck approach provides a powerful framework for extracting noise-invariant features for speaker verification. By directly optimizing for noise invariance rather than signal reconstruction, the system achieves state-of-the-art performance in both noisy and clean conditions. Future work will focus on normalizing feature dimensions for more direct comparisons and applying this technique to broader audio processing tasks.
Summary updated