Abstract
The AISPEECH-SJTU team secured first place in the 2020 Accented English Speech Recognition Challenge (AESRC) by rethinking the representation and augmentation of accented speech. Their system leverages Phone Posteriorgrams (PPGs) to isolate phonetic patterns from speaker-specific noise and introduces a novel TTS-based data augmentation strategy. This approach resulted in an 83.63% average accuracy, outperforming the runner-up by over 11%.
Introduction: The Challenge of Accent Identification
Identifying accents is a cornerstone of robust speech technology, yet it remains notoriously difficult. Traditional acoustic features like Filter Banks (FBANKs) often capture "nuisance" variables—such as a speaker’s unique voice or background noise—rather than the underlying phonetic patterns that define an accent.
Furthermore, deep learning models are data-hungry, but accented datasets are often limited. The AESRC 2020 provided only 160 hours of data across eight countries (US, UK, China, India, Japan, Korea, Portugal, and Russia). To build a system that generalizes well, the AISPEECH-SJTU team focused on extracting "pure" accent features and artificially expanding the training set without losing regional nuances.
Methodology
The team employed a multi-pronged strategy to dominate the leaderboard, centered on robust feature extraction and massive data synthesis.
- ASR-based PPG Features: Instead of raw audio frequencies, the system uses Phone Posteriorgrams (PPGs). By training an Automatic Speech Recognition (ASR) model on a massive combined dataset (including Librispeech), the team extracted vectors representing the probability of specific phonetic classes. PPGs focus on phonetic content, making them naturally speaker-independent and ideal for isolating accent-specific pronunciation.
- TTS-based Data Augmentation: The team used Text-to-Speech (TTS) to generate 4,800 hours of synthetic speech. They fine-tuned FastSpeech synthesizers on each of the eight target accents and used LPCNet as a vocoder. This provided the model with significant variability in text and speaker styles while maintaining the target accent.
- Hierarchical Multi-Embedding Joint Model: The final classifier fused two architectures: a standard Time Delay Neural Network (TDNN) and a RES2SETDNN (incorporating Res2Net and Squeeze-and-Excitation modules).
- Test-Time Augmentation (TTA): During inference, multiple versions of test clips were created at different speeds (tempo perturbation) and aggregated to boost final accuracy.
Key Results and Ablation Study
The system achieved an average accuracy of 83.63% on the evaluation set, ranking 1st in the challenge and outperforming the baseline by nearly 19%.
As shown in the ablation study, the transition from FBANK features to PPGs was the single most significant contributor to performance. For the TDNN model, accuracy jumped from 60.24% to 83.74% simply by changing the feature representation. Adding TTS-based augmentation further pushed these gains, demonstrating that synthetic data carries enough accent-specific information to improve classifier robustness.
Insights and Conclusion
A key takeaway from this work is the superiority of phonetics over physics in accent tasks; accents are fundamentally about phonetic shifts rather than raw spectral energy. Additionally, t-SNE visualizations of the embeddings revealed that while some accents (like Indian English) are highly distinct, others (like US English) show significant overlap, suggesting that "General American" often serves as a phonetic baseline.
While the system set a new benchmark, a 7% performance gap between development and evaluation sets highlights the ongoing challenge of "in-the-wild" speech variability. Future research will likely explore advanced cross-accent voice conversion to further refine synthetic training data.
Summary updated