Aipnet: Generative Adversarial Pre-Training of Accent-Invariant Networks for End-To-End Speech Recognition

Yi-Chen Chen, Zhaojun Yang, Ching-Feng Yeh, Mahaveer Jain, Michael L. SeltzerICASSP 2020 · DOI

TL;DR

This paper introduces AIPNet, a novel generative adversarial pre-training framework designed to create accent-invariant speech representations for end-to-end Automatic Speech Recognition (ASR). By disentangling accent-specific characteristics from linguistic content, AIPNet significantly improves ASR robustness across diverse accents, even when transcriptions for some accents are unavailable.

What Problem Does This Paper Address?

Accents are a major source of variability in spoken language, posing a significant challenge for robust Automatic Speech Recognition (ASR) systems. An ASR system trained on one accent often performs poorly on speech from a different accent. While existing approaches include training on all available accent data (accent-independent) or incorporating accent-specific information (accent-dependent), neither fully solves the problem of building a universally robust ASR system. This paper aims to bridge this gap by developing a unified, accent-independent ASR system that generalizes well across various accents, particularly in scenarios where labeled data for all accents is scarce.

Methods

The core of the proposed solution is the AIPNet (Accent Invariant Pre-training Networks) framework, which operates in two stages: pre-training and fine-tuning.

Pre-training: This stage focuses on learning accent-invariant representations using Generative Adversarial Nets (GANs).

  • Accent-Invariant and Accent-Specific Generators: Two generators, $G_{AI}$ and $G_{AS}$, are trained to disentangle acoustic features into accent-invariant and accent-specific components, respectively.
  • Discriminators: Two discriminators, $D_{AI}$ and $D_{AS}$, are used to distinguish between real and generated accent-specific and accent-invariant features. $D_{AI}$ tries to identify the accent from $G_{AI}$'s output, while $G_{AI}$ is trained to fool $D_{AI}$, thus learning to produce accent-agnostic representations.
  • Reconstruction Loss: A decoder is trained to reconstruct the original acoustic features from both $G_{AI}$ and $G_{AS}$ outputs, ensuring that important acoustic information is preserved.
  • Consistency Regularization: This loss encourages the accent-specific representations from $G_{AS}$ to remain consistent across time frames within an utterance, further stabilizing accent-specific information.
  • Adversarial Training: The pre-training process involves an iterative min-max game between the discriminators and generators, aiming to achieve accent-invariance. Crucially, this pre-training stage does not require transcriptions, making it highly valuable for leveraging large amounts of unlabeled accented speech data.

Fine-tuning: The learned accent-invariant embeddings from $G_{AI}$ are then plugged into a standard end-to-end ASR model, specifically an attention-based encoder-decoder (Listen, Attend and Spell - LAS) model. This fine-tuning stage optimizes the combined model for the ASR task, further refining the accent-invariant representations for speech recognition.

Datasets

The experiments were conducted on a dataset containing utterances from 9 English accents: United States (US), Korea (KR), Philippines (PH), Canada (CA), India (IN), France (FR), Britain (GB), Vietnam (VN), and Latin America (LA). The dataset comprised 4 million utterances (3.8K hours) for training, with a validation split. The testing set contained 10.8K utterances. Acoustic features were extracted using 80-dimensional log Mel-filterbank energies.

Key Insights and Ideas

  • Decoupling Accent from Content: The core idea of separating accent-specific information from linguistic content through adversarial training is a powerful mechanism for achieving accent invariance.
  • Leveraging Unlabeled Data: The pre-training stage's independence from transcriptions is a significant advantage, allowing the model to learn from vast amounts of readily available accented speech data that might otherwise be unusable for supervised ASR training.
  • Pre-training for Robustness: Pre-training accent-invariant representations before fine-tuning on the ASR task provides a strong foundation for robustness, leading to better generalization across unseen accents.
  • Semi-supervised Learning Effectiveness: The paper demonstrates that AIPNet is particularly effective in semi-supervised settings where transcriptions are only available for a subset of accents (e.g., US accent), highlighting its practical utility.
  • Visualization of Accent Invariance: t-SNE visualizations of the learned embeddings clearly show that AIPNet's representations are mixed across accents, unlike baseline models where embeddings tend to cluster by accent.

Key Results

  • Supervised Setting: AIPNet achieved a relative WER reduction of 2.3-4.5% on average compared to baselines when transcriptions were available for all accents.
  • Semi-supervised Setting: In scenarios with limited transcriptions (only US accent), AIPNet significantly outperformed the baseline, achieving 3.4-11.3% relative WER reduction across all accents.
  • With Pseudo-Labeling: When pseudo-labeling was applied to unlabeled data, AIPNet (specifically F2) showed a further relative WER reduction of 1.6-6.1% on average, consistently performing best in most non-US accents.
  • Reduced Overfitting: The analysis suggests that the accent-related regularization in AIPNet's fine-tuning objective helps prevent overfitting, leading to better generalization, especially in semi-supervised settings.

Limitations and Future Work

  • Hyperparameter Sensitivity: While the authors state that results are generally stable across hyperparameter settings, fine-tuning specific hyperparameters (λ1, λ2, λ3, λ4) still requires careful tuning.
  • Supervised vs. Semi-supervised Performance: The paper notes that the regularization benefits of AIPNet are more pronounced in semi-supervised settings. In fully supervised settings with abundant data, the additional regularization might slightly hinder performance compared to a purely supervised model, suggesting that the optimal fine-tuning strategy might depend on data availability.
  • Generalizability to Other Variabilities: The authors suggest that AIPNet's framework for learning domain-invariance can be easily generalized to model other variabilities like speaker identity or speech noise, which could be a promising direction for future research.
  • Further Exploration of Fine-tuning: The paper mentions two fine-tuning strategies and reports results for both. Further investigation into the optimal fine-tuning approach for different data scenarios could be beneficial.

Summary updated

BibTeX
@inproceedings{chen2020aipnet,
  author = {Yi-Chen Chen and Zhaojun Yang and Ching-Feng Yeh and Mahaveer Jain and Michael L. Seltzer},
  title = {{Aipnet: Generative Adversarial Pre-Training of Accent-Invariant Networks for End-To-End Speech Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  doi = {10.1109/icassp40776.2020.9053098},
}
Figure