Adversarial Multi-Task Learning of Deep Neural Networks for Robust Speech Recognition

Yusuke ShinoharaInterspeech 2016

TL;DR

This paper introduces a novel "adversarial multi-task learning" approach for deep neural networks (DNNs) to improve speech recognition robustness against noise. By jointly training a primary speech recognition task with a secondary noise-condition classification task in an adversarial manner, the model learns representations that are both discriminative for speech and invariant to noise, leading to better recognition performance in noisy environments.

What Problem Does This Paper Address?

Speech recognition systems often struggle with noise, which can significantly degrade performance. While deep neural networks (DNNs) have shown remarkable noise robustness, this is largely attributed to their ability to learn representations that are inherently invariant to noise, especially in higher layers. However, there's been a lack of explicit methods to enhance this noise invariance. The authors aim to bridge this gap by proposing a technique that actively encourages the DNN to learn representations that are less sensitive to varying noise conditions, thereby boosting overall robustness.

Methods

The core innovation is adversarial multi-task learning. This approach extends standard multi-task learning by introducing a "gradient reversal layer" (GRL). Here's how it works:

  • Primary Task: Standard senone classification (predicting the phonetic units of speech).
  • Secondary Task: Domain (noise condition) classification.
  • Shared Representation: A common input sub-network extracts features for both tasks.
  • Adversarial Component: The GRL is inserted between the shared feature extractor and the secondary task classifier. During training, the GRL acts as an identity function in the forward pass but reverses the gradient in the backward pass. This means the shared feature extractor is trained to minimize the accuracy of the domain classifier (i.e., make it hard for the network to tell which noise condition the speech came from), while simultaneously maximizing the accuracy of the primary senone classification task.

Essentially, the network is penalized for learning noise-specific features, forcing it to focus on speech-discriminative features that are independent of the noise. After training, the secondary task classifier is discarded, leaving a robust DNN for speech recognition.

Datasets

  • Training: A noise-corrupted version of the Wall Street Journal (WSJ) dataset was created. This involved adding one of four noise types (car, exhibition booth, station, crossing) at one of five Signal-to-Noise Ratio (SNR) levels (5, 10, 15, 20, 100 dB) to the original WSJ training utterances.
  • Evaluation: 12 evaluation sets were created using the same four noise types and five SNR levels. Crucially, these included eight "unknown" noise types not seen during training, allowing for a robust evaluation of generalization.

Key Insights and Ideas

  • Explicitly Enhancing Invariance: The paper moves beyond relying on implicit noise invariance learned by standard DNNs. By directly training against a noise classifier, the method explicitly forces the network to discard noise-dependent information.
  • The Power of Adversarial Training for Robustness: The adversarial setup is shown to be effective not just for unsupervised domain adaptation but also for supervised tasks like robust speech recognition, where labeled data for different conditions is available.
  • Domain-Invariant Representations are Key: The success hinges on the idea that by making representations "domain-invariant" (i.e., insensitive to the noise condition), the primary task (speech recognition) becomes more robust.

Key Results

The proposed adversarial multi-task learning (AMT) DNN achieved better Word Error Rates (WERs) than standard single-task (ST) DNNs for nine out of twelve evaluation sets. On average, the AMT-DNN resulted in a 2.04% relative error rate reduction compared to the ST-DNN. While some noise types showed more significant improvements than others, the overall trend indicates the effectiveness of the adversarial approach.

Limitations and Future Work

  • Unclear Improvement Mechanisms: The authors note that they couldn't pinpoint exactly why improvements varied across different noise types and suggest further study is needed to understand these dynamics.
  • Tuning Complexity: Adversarial training can be notoriously difficult to tune. The paper mentions that more dedicated algorithms and tuning might further enhance performance.
  • Broader Applicability: The authors suggest that this framework is not limited to noise robustness and could be applied to improve robustness against other variations like reverberation, channel effects, or speaker characteristics.

Summary updated

BibTeX
@inproceedings{shinohara2016adversarial,
  author = {Yusuke Shinohara},
  title = {{Adversarial Multi-Task Learning of Deep Neural Networks for Robust Speech Recognition}},
  booktitle = {Interspeech},
  year = {2016},
}
Figure