REDAT: Accent-Invariant Representation for End-to-End ASR by Domain Adversarial Training with Relabeling

Hu Hu, Xuesong Yang, Zeynab Raeesy, Jinxi Guo, Gökçe Keskin, Harish Arsikere, Ariya Rastrow, Andreas Stolcke, Roland Maas

Introduction

End-to-end Automatic Speech Recognition (ASR) systems, such as those powering modern smart speakers, often exhibit performance degradation when encountering accents that differ from the "native" training data. While building accent-specific models is one solution, it lacks scalability and fails to address "unseen" accents—regions or dialects not represented in the training set.

The core challenge lies in developing accent-invariant representations: internal features that capture linguistic content while remaining agnostic to the speaker's accent. While Domain Adversarial Training (DAT) is a common approach for achieving domain invariance, its application in ASR has often relied on rigid geographic labels (e.g., "US" vs. "UK") that fail to capture the acoustic nuances of diverse speakers. This paper introduces reDAT, which combines a mathematically grounded DAT framework with a novel relabeling strategy to improve robustness across both seen and unseen accents.

Methodology

The researchers utilize a Recurrent Neural Network Transducer (RNN-T) architecture, enhanced by two primary innovations: a theoretical foundation for DAT and the reDAT relabeling process.

Theoretical Foundation

The authors provide a mathematical proof that performing gradient reversal in adversarial training is equivalent to minimizing the Jensen-Shannon Divergence (JSD) between different accent distributions. This proof transitions DAT from a heuristic technique to a grounded approach for forcing the model to generate identical internal "fingerprints" for different accents.

reDAT: Relabeling for Acoustic Nuance

Standard DAT typically uses "hard" geographic labels. However, accents are fluid; a non-native speaker in one region may sound more similar to a speaker in another region than to a native local. To address this, the authors introduced two relabeling techniques:

  1. Unsupervised Clustering: An accent classifier groups similar-sounding utterances into $k$ clusters, creating granular acoustic labels.
  2. Soft Labels: Instead of binary labels, a probability distribution is used to capture the nuances of non-native speech.
Overview of the reDAT architecture and relabeling process.

As shown in the architecture diagram, a "Generator" (part of the ASR encoder) attempts to create features that an "Accent Classifier" cannot distinguish. This adversarial process strips away accent-specific information, leaving only core linguistic content for the RNN-T decoder.

Experimental Setup

The study leveraged a massive dataset of 23,000 hours of de-identified human-labeled speech from Amazon Alexa devices. The training set included 13K hours of US English (en-US), 6K hours of British English (en-GB), and 4K hours of Indian English (en-IN). Evaluation was conducted on native and non-native speakers of en-US and en-GB, with Australian English (en-AU) serving as a completely "unseen" accent to test generalization.

Results and Discussion

The experimental results demonstrate that reDAT significantly outperforms standard data pooling and basic DAT approaches, particularly for challenging speaker groups.

Performance comparison across various accent categories and models.

Key findings include:

  • Unseen Accents: The adversarial approach achieved up to a 13% relative reduction in Word Error Rate (WER) on the unseen Australian accent compared to standard data pooling.
  • Non-Native Performance: The reDAT-soft approach yielded a 3% to 8% improvement over standard DAT for non-native speakers of American and British English, highlighting the benefit of capturing acoustic fluidity.
  • No "Native Tax": Crucially, the gains in robustness did not come at the expense of native speech accuracy; performance on standard en-US and en-GB test sets remained highly competitive.

Conclusion

By proving that DAT minimizes JSD and introducing acoustic-based relabeling, the authors have developed a more robust framework for accent-invariant ASR. While choosing the optimal number of clusters ($k$) remains a challenge, the success of reDAT suggests that moving beyond geographic labels is essential for truly global voice recognition. Future work may explore automated granularity selection or applying these techniques to other domain-mismatch scenarios like background noise or microphone variability.

Summary updated

BibTeX
@inproceedings{hu2021redat,
  author = {Hu Hu and Xuesong Yang and Zeynab Raeesy and Jinxi Guo and Gökçe Keskin and Harish Arsikere and Ariya Rastrow and Andreas Stolcke and Roland Maas},
  title = {{REDAT: Accent-Invariant Representation for End-to-End ASR by Domain Adversarial Training with Relabeling}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9414291},
}
Figure