Adversarial Learning of Raw Speech Features for Domain Invariant Speech Recognition

Aditay Tripathi, Aanchan Mohan, Saket Anand, Maneesh Singh

Introduction

Automatic Speech Recognition (ASR) systems frequently encounter "domain shift"—a significant drop in accuracy when the acoustic characteristics of test data (the target domain) diverge from the training data (the source domain). These shifts are typically driven by variations in speaker gender, accents, or environmental noise.

Traditional ASR pipelines rely on hand-crafted features like Mel-Frequency Cepstral Coefficients (MFCCs). However, these features may inadvertently preserve domain-specific information that hinders generalization. This paper proposes a method for unsupervised domain adaptation by learning robust features directly from raw audio waveforms using adversarial training, eliminating the need for expensive labeled data in the target domain.

Methodology: The Domain Adversarial Neural Network

The researchers employ a Domain Adversarial Neural Network (DANN) architecture, characterized by a tripartite "Y-shaped" structure:

  1. Feature Extractor ($G_f$): A multi-layer Convolutional Neural Network (CNN) that processes raw speech windows (310ms context) to extract high-level representations.
  2. Label (Senone) Classifier ($G_y$): Predicts the phonetic state (senone) of the speech, encouraging the extraction of descriptive features for recognition.
  3. Domain Classifier ($G_d$): Attempts to identify the domain of the input (e.g., speaker gender or accent).
Architecture of the Domain Adversarial Neural Network (DANN) for raw speech.

The critical innovation is the Gradient Reversal Layer (GRL) situated between the feature extractor and the domain classifier. During backpropagation, the GRL multiplies the gradients from the domain classifier by a negative constant ($-\lambda$). This adversarial setup forces the feature extractor to minimize the label classification error while maximizing the domain classification error. Consequently, the network learns to strip away domain-specific traits, retaining only domain-invariant features essential for speech recognition.

Experimental Setup

The model was evaluated on two distinct speech corpora to test its robustness against common sources of variability:

  • TIMIT (Gender Variability): The model was trained on male speakers (source) and adapted to female speakers (target) using unlabeled data.
  • Voxforge (Accent Variability): Cross-domain adaptation was performed between American and British accents.

Key Results and Analysis

The adversarial approach consistently outperformed standard neural networks (NN) without adaptation across both datasets.

Gender Adaptation (TIMIT)

In the gender shift experiment, a standard model trained on male speech and tested on female speech yielded a Phone Error Rate (PER) of 37.20%. By applying DANN, the error rate dropped to 32.26%, representing a substantial ~5% absolute improvement.

Phone Error Rate (PER) results for gender adaptation on the TIMIT dataset.

Accent Adaptation (Voxforge)

Similar gains were observed for accent variability. When adapting from American (source) to British (target), the Word Error Rate (WER) decreased from 28.11% to 23.10%. Conversely, moving from British to American saw the WER drop from 23.37% to 23.16%.

Raw Speech vs. MFCCs

A key finding is that raw speech features are more amenable to adversarial training than traditional MFCCs. In the gender shift task, using DANN with MFCCs only reduced the error by 2.45%, whereas raw speech yielded nearly double that improvement. This suggests that raw waveforms provide the adversarial process with more "room" to filter out domain-specific nuances.

Conclusion and Future Work

This research demonstrates that ASR models can learn to ignore "distractions" like gender and accents by training directly on raw audio. While the model achieves impressive unsupervised adaptation, it faces an "invariance trade-off": improvements in the target domain often come at the cost of a slight performance dip in the source domain.

Future research could explore maintaining source accuracy during adaptation, investigating more complex shifts like high-noise environments, or utilizing deeper architectures such as ResNets or CLDNNs to further refine the quality of learned features.

Summary updated

Abstract

Recent advances in neural network based acoustic modelling have shown significant improvements in automatic speech recognition (ASR) performance. In order for acoustic models to be able to handle large acoustic variability, large amounts of labeled data is necessary, which are often expensive to obtain. This paper explores the application of adversarial training to learn features from raw speech that are invariant to acoustic variability. This acoustic variability is referred to as a domain shift in this paper. The experimental study presented in this paper leverages the architecture of Domain Adversarial Neural Networks (DANNs) [1] which uses data from two different domains. The DANN is a Y-shaped network that consists of a multi-layer CNN feature extractor module that is common to a label (senone) classifier and a so-called domain classifier. The utility of DANNs is evaluated on multiple datasets with domain shifts caused due to differences in gender and speaker accents. Promising empirical results indicate the strength of adversarial training for unsupervised domain adaptation in ASR, thereby emphasizing the ability of DANNs to learn domain invariant features from raw speech.
BibTeX
@inproceedings{tripathi2018adversarial,
  author = {Aditay Tripathi and Aanchan Mohan and Saket Anand and Maneesh Singh},
  title = {{Adversarial Learning of Raw Speech Features for Domain Invariant Speech Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=Lfu7GPnIMYT},
}
Figure