AISpeech-SJTU Accent Identification System for the Accented English Speech Recognition Challenge

Houjun Huang, Xu Xiang, Yexin Yang, Rao Ma, Yanmin Qian

Abstract

The AISPEECH-SJTU team secured first place in the 2020 Accented English Speech Recognition Challenge (AESRC) by rethinking the representation and augmentation of accented speech. Their system leverages Phone Posteriorgrams (PPGs) to isolate phonetic patterns from speaker-specific noise and introduces a novel TTS-based data augmentation strategy. This approach resulted in an 83.63% average accuracy, outperforming the runner-up by over 11%.

Introduction: The Challenge of Accent Identification

Identifying accents is a cornerstone of robust speech technology, yet it remains notoriously difficult. Traditional acoustic features like Filter Banks (FBANKs) often capture "nuisance" variables—such as a speaker’s unique voice or background noise—rather than the underlying phonetic patterns that define an accent.

Furthermore, deep learning models are data-hungry, but accented datasets are often limited. The AESRC 2020 provided only 160 hours of data across eight countries (US, UK, China, India, Japan, Korea, Portugal, and Russia). To build a system that generalizes well, the AISPEECH-SJTU team focused on extracting "pure" accent features and artificially expanding the training set without losing regional nuances.

Methodology

The team employed a multi-pronged strategy to dominate the leaderboard, centered on robust feature extraction and massive data synthesis.

Overview of the AISPEECH-SJTU accent identification system pipeline, including PPG extraction and TTS augmentation.
  1. ASR-based PPG Features: Instead of raw audio frequencies, the system uses Phone Posteriorgrams (PPGs). By training an Automatic Speech Recognition (ASR) model on a massive combined dataset (including Librispeech), the team extracted vectors representing the probability of specific phonetic classes. PPGs focus on phonetic content, making them naturally speaker-independent and ideal for isolating accent-specific pronunciation.
  2. TTS-based Data Augmentation: The team used Text-to-Speech (TTS) to generate 4,800 hours of synthetic speech. They fine-tuned FastSpeech synthesizers on each of the eight target accents and used LPCNet as a vocoder. This provided the model with significant variability in text and speaker styles while maintaining the target accent.
  3. Hierarchical Multi-Embedding Joint Model: The final classifier fused two architectures: a standard Time Delay Neural Network (TDNN) and a RES2SETDNN (incorporating Res2Net and Squeeze-and-Excitation modules).
  4. Test-Time Augmentation (TTA): During inference, multiple versions of test clips were created at different speeds (tempo perturbation) and aggregated to boost final accuracy.

Key Results and Ablation Study

The system achieved an average accuracy of 83.63% on the evaluation set, ranking 1st in the challenge and outperforming the baseline by nearly 19%.

Ablation study showing the performance gains from switching to PPG features and adding various augmentation techniques.

As shown in the ablation study, the transition from FBANK features to PPGs was the single most significant contributor to performance. For the TDNN model, accuracy jumped from 60.24% to 83.74% simply by changing the feature representation. Adding TTS-based augmentation further pushed these gains, demonstrating that synthetic data carries enough accent-specific information to improve classifier robustness.

Insights and Conclusion

A key takeaway from this work is the superiority of phonetics over physics in accent tasks; accents are fundamentally about phonetic shifts rather than raw spectral energy. Additionally, t-SNE visualizations of the embeddings revealed that while some accents (like Indian English) are highly distinct, others (like US English) show significant overlap, suggesting that "General American" often serves as a phonetic baseline.

While the system set a new benchmark, a 7% performance gap between development and evaluation sets highlights the ongoing challenge of "in-the-wild" speech variability. Future research will likely explore advanced cross-accent voice conversion to further refine synthetic training data.

Summary updated

Abstract

This paper describes the AISpeech-SJTU system for the accent identification track of the Interspeech-2020 Accented English Speech Recognition Challenge. In this challenge track, only 160-hour accented English data collected from 8 countries and the auxiliary Librispeech dataset are provided for training. To build an accurate and robust accent identification system, we explore the whole system pipeline in detail. First, we introduce the ASR based phone posteriorgram (PPG) feature to accent identification and verify its efficacy. Then, a novel TTS based approach is carefully designed to augment the very limited accent training data for the first time. Finally, we propose the test time augmentation and embedding fusion schemes to further improve the system performance. Our final system is ranked first in the challenge and outperforms all the other participants by a large margin. The submitted system achieves 83.63% average accuracy on the challenge evaluation data, ahead of the others by more than 10% in absolute terms.
BibTeX
@inproceedings{huang2021aispeech,
  author = {Houjun Huang and Xu Xiang and Yexin Yang and Rao Ma and Yanmin Qian},
  title = {{AISpeech-SJTU Accent Identification System for the Accented English Speech Recognition Challenge}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=0pVFqjBHJ1},
}
Figure