Training Multi-task Adversarial Network for Extracting Noise-robust Speaker Embedding

Jianfeng Zhou, Tao Jiang, Lin Li, Qingyang Hong, Zhe Wang, Bing-yin Xia2019 · DOI

TL;DR

This paper introduces a novel Multi-Task Adversarial Network (MTAN) to create speaker embeddings that are robust to noisy environments. By training an encoder to fool a noise-type discriminator, the network learns representations that are both speaker-discriminative and noise-invariant, leading to significant improvements in speaker verification accuracy.

What Problem Does This Paper Address?

Speaker recognition systems, particularly those for verifying a person's identity from their voice, struggle significantly in the presence of background noise. Traditional methods often rely on techniques like i-vectors or deep learning models that are trained on clean data, leading to a sharp performance drop when encountering real-world noisy conditions. While some approaches focus on speech enhancement or training with noisy data, this paper aims to directly learn noise-robust speaker embeddings by leveraging adversarial training. The core challenge is to disentangle speaker identity from the interfering noise, creating a representation that is stable regardless of the acoustic environment.

Methods

The proposed framework, MTAN, consists of three main components:

  • Encoder: A CNN-based architecture that extracts utterance-level speaker embeddings. This is the core component responsible for learning the representation.
  • Classifier: A standard classifier that uses the speaker embeddings to identify the speaker. This component is trained cooperatively with the encoder to ensure speaker discriminability.
  • Discriminator: This component is trained to identify the type of noise present in the speaker embeddings. Crucially, the encoder is trained adversarially against this discriminator. The goal is to make the encoder produce embeddings that "fool" the discriminator, meaning the noise type cannot be reliably identified from the embedding.

The paper also introduces two novel loss functions for the adversarial training:

  • FL-Loss (Fixed Label Loss): This loss regularizes the encoder by forcing it to produce embeddings similar to those from "clean" speech, effectively mapping noisy data distributions towards a clean one.
  • AL-Loss (Anti-Label Loss): This loss, inspired by FL-Loss, uses an "anti-label" (the inverse of the ground truth noise label) to train the encoder. This encourages the encoder to produce embeddings that are equally likely to be classified as any noise type, thus making them invariant to noise.

A key innovation is a training stability strategy that uses the training accuracy of the noise discriminator as an indicator. By monitoring this accuracy, the authors adjust the adversarial loss weights to maintain a balance between the encoder and discriminator, preventing one from overpowering the other and ensuring effective adversarial learning.

Datasets

The experiments were conducted on two publicly available datasets:

  • Aishell-1: A Mandarin corpus used for training and testing.
  • Librispeech: An English corpus used for training and testing.

For training, the authors artificially added various types of noise (white, babble, mensa, cafeteria, callcener) at different Signal-to-Noise Ratios (SNRs) to the original clean datasets. The test sets were also corrupted with noise at various SNRs.

Key Insights and Ideas

  • Adversarial Training for Noise Robustness: The core insight is that by training an encoder to fool a noise-type discriminator, we can force it to learn representations that are inherently robust to noise. This is a more direct approach than traditional speech enhancement.
  • Disentangling Speaker and Noise: The adversarial setup effectively forces the network to disentangle the speaker's identity from the noise characteristics present in the audio.
  • AL-Loss for Noise Invariance: The proposed AL-Loss is particularly interesting as it aims for a more complete noise invariance by making embeddings equally likely to be misclassified as any noise type.
  • Training Stability is Crucial: The paper highlights that adversarial training can be unstable. Their proposed accuracy-based monitoring and adjustment strategy is vital for achieving good performance.
  • Benefits in Clean Conditions Too: Surprisingly, the MTAN framework also shows improved performance even in clean conditions, suggesting that the learned representations are generally more discriminative.

Key Results

The proposed MTAN framework, particularly when using the AL-Loss and FL-Loss, significantly outperformed baseline methods (CNN trained on clean data, and CNN trained on mixed noisy data) in terms of Equal Error Rate (EER) across various noise types and SNRs on both Librispeech and Aishell-1 corpora.

  • For example, on Librispeech, the MTAN with AL-Loss achieved EERs as low as 5.89% in noisy conditions, compared to 16.58% for the MIX system at 10dB SNR for white noise.
  • The fusion of FL and AL losses further improved performance.
  • Even in clean conditions, MTAN outperformed the baseline and MIX systems.

Limitations and Future Work

  • Limitations: The paper acknowledges that the encoder is trained better than the discriminator in their experiments, leading them to set a lower threshold for the training stability strategy. This suggests a potential imbalance that might need further investigation. The current work focuses on specific noise types; generalizing to a wider range of unknown noises could be a challenge.
  • Future Work: The authors plan to conduct experiments in lower SNR conditions and explore other related applications. They also suggest investigating the fusion of FL and AL losses more deeply.

Summary updated

Abstract

Under noisy environments, to achieve the robust performance of speaker recognition is still a challenging task. Motivated by the promising performance of multi-task training in a variety of image processing tasks, we explore the potential of multitask adversarial training for learning a noise-robust speaker embedding. In this paper, we present a novel framework that consists of three components: an encoder that extracts the noise-robust speaker embeddings; a classifier that classifies the speakers; a discriminator that discriminates the noise type of the speaker embeddings. Additionally, we propose a training strategy using the training accuracy as an indicator to stabilize the multi-class adversarial optimization process. We conduct our experiments on the English and Mandarin corpuses and the experimental results demonstrate that our proposed multi-task adversarial training method could greatly outperform the other methods without adversarial training in noisy environments. Furthermore, the experiments indicate that our method is also able to improve the speaker verification performance under the clean condition.
BibTeX
@misc{zhou2019training,
  author = {Jianfeng Zhou and Tao Jiang and Lin Li and Qingyang Hong and Zhe Wang and Bing-yin Xia},
  title = {{Training Multi-task Adversarial Network for Extracting Noise-robust Speaker Embedding}},
  year = {2019},
  doi = {10.1109/icassp.2019.8683828},
}
Figure