Adversarial Network Bottleneck Features for Noise Robust Speaker Verification

Hong Yu, Zheng-Hua Tan, Zhanyu Ma, Jun Guo

Abstract

Speaker Verification (SV) systems are highly sensitive to environmental noise. This paper introduces an Adversarial Network Bottleneck (AN-BN) feature extraction framework designed to produce noise-invariant representations. By training an encoding network to deceive a noise-type discriminator, the system learns to suppress noise-specific patterns. Experimental results on the RSR2015 dataset demonstrate that AN-BN features significantly outperform traditional and deep-learning-based speech enhancement methods, even improving performance in clean-speech scenarios.

Introduction

Confirming a person's identity via voice—Speaker Verification—is a critical task for secure mobile and remote applications. However, real-world performance is often degraded by additive noise such as babble, traffic, or electronic interference. Traditional front-end solutions like Wiener filters or spectral subtraction often introduce artifacts that hinder downstream verification. This research proposes moving beyond signal reconstruction toward learning features that are inherently robust to environmental variations.

Methodology: The Adversarial Framework

The proposed system utilizes an Adversarial Network Bottleneck (AN-BN) architecture. It consists of two primary components: an Encoding Network (EN) and a Discriminative Network (DN).

The AN-BN architecture: An Encoding Network (EN) produces bottleneck features, while a Discriminative Network (DN) attempts to identify the noise type.

The EN takes 11 frames of Mel-frequency cepstral coefficients (MFCCs) and maps them to a 128-dimensional bottleneck layer. The DN is trained to classify the noise type (e.g., White, Babble, Market) present in these features. In an adversarial setup, the EN is trained with a "deceptive" objective: it must produce features that the DN classifies as "clean," regardless of the actual input noise. This competition forces the EN to strip away noise-specific information, retaining only the underlying speech characteristics.

Experimental Setup

The framework was evaluated using the RSR2015 database (male speakers). The Universal Background Model (UBM) was trained on the TIMIT dataset. Five noise types were considered: Babble, Cantine, Market, Airplane, and White Gaussian noise. To simulate realistic conditions, training and testing were conducted using different mobile devices (Samsung Nexus, Samsung Galaxy S, and HTC Desire) to account for microphone variability.

Results and Discussion

The AN-BN features were compared against several baselines: standard MFCCs (No Enh.), statistical enhancement (STSA-MMSE), and deep neural network-based speech enhancement (DNN-SE).

Equal Error Rate (EER) results across different noise types and SNRs, comparing the proposed AN-BN features (NS-AN, NG-AN) against baseline methods.

As shown in the results table, the AN-BN approach (labeled NS-AN and NG-AN) consistently achieves the lowest Equal Error Rates (EER). Key observations include:

  • Robustness to High Noise: In low SNR environments (0dB to 5dB), the proposed features reduced EER by approximately 25% for White and Babble noise and up to 40% for other types compared to DNN-SE.
  • Clean Condition Gains: A notable finding was the improvement in clean-speech performance. By training the EN on clean speech from multiple sessions with a single "clean" label, the network learned to reduce session-to-session variability, a common challenge in SV.
  • Generalization: The system demonstrated strong performance on SNRs not explicitly seen during the training phase, indicating high generalization capability.

Conclusion

The Adversarial Network Bottleneck framework offers a powerful method for extracting noise-invariant features for speaker verification. By focusing on invariance rather than signal reconstruction, the system achieves state-of-the-art robustness across diverse environmental conditions. Future research will explore the applicability of these features to Automatic Speech Recognition (ASR) and investigate dimensionality reduction techniques to optimize computational efficiency.

Summary updated

Abstract

In this paper, we propose a noise robust bottleneck feature representation which is generated by an adversarial network (AN). The AN includes two cascade connected networks, an encoding network (EN) and a discriminative network (DN). Mel-frequency cepstral coefficients (MFCCs) of clean and noisy speech are used as input to the EN and the output of the EN is used as the noise robust feature. The EN and DN are trained in turn, namely, when training the DN, noise types are selected as the training labels and when training the EN, all labels are set as the same, i.e., the clean speech label, which aims to make the AN features invariant to noise and thus achieve noise robustness. We evaluate the performance of the proposed feature on a Gaussian Mixture Model-Universal Background Model based speaker verification system, and make comparison to MFCC features of speech enhanced by short-time spectral amplitude minimum mean square error (STSA-MMSE) and deep neural network-based speech enhancement (DNN-SE) methods. Experimental results on the RSR2015 database show that the proposed AN bottleneck feature (AN-BN) dramatically outperforms the STSA-MMSE and DNN-SE based MFCCs for different noise types and signal-to-noise ratios. Furthermore, the AN-BN feature is able to improve the speaker verification performance under the clean condition.
BibTeX
@inproceedings{yu2017adversarial,
  author = {Hong Yu and Zheng-Hua Tan and Zhanyu Ma and Jun Guo},
  title = {{Adversarial Network Bottleneck Features for Noise Robust Speaker Verification}},
  booktitle = {Interspeech},
  year = {2017},
  doi = {10.21437/interspeech.2017-883},
}
Figure