Abstract
Automatic identification of foreign accents is valuable for many speech systems, such as speech recognition, speaker identification, voice conversion, etc. The INTERSPEECH 2016 Native Language Sub-Challenge is to identify the native languages of non-native English speakers from eleven countries. Since differences in accent are due to both prosodic and articulation characteristics, a combination of long-term and short-term training is proposed in this paper. Each speech sample is processed into multiple speech segments with equal length. For each segment, deep neural networks (DNNs) are used to train on long-term statistical features, while recurrent neural networks (RNNs) are used to train on short-term acoustic features. The result for each speech sample is calculated by linearly fusing the results from the two sets of networks on all segments. The performance of the proposed system greatly surpasses the provided baseline system. Moreover, by fusing the results with the baseline system, the performance can be further improved.
Introduction
Identifying a speaker's native language from their foreign-accented English is a complex task because accents manifest through both articulation (short-term sound production) and prosody (long-term rhythm, stress, and intonation). Most existing systems struggle by focusing on only one of these scales. Furthermore, real-world speech data often contains background noise and non-fluent fillers, complicating the identification process. Improving accent identification is crucial for robust voice assistants, forensic profiling, and voice conversion technologies.
Methods
The researchers proposed a multi-scale approach that processes speech at two different time scales to capture both prosodic and articulatory signatures.
- Preprocessing: Voice Activity Detection (VAD) is used to remove silences longer than 300ms. The remaining audio is divided into 4-second segments.
- Long-Term Path (DNN): For each segment, 6,373 statistical features (e.g., mean and standard deviation of pitch and energy) are extracted. A DNN with three hidden layers is trained on these features to capture prosodic patterns.
- Short-Term Path (RNN): Within the same segments, 25ms windows are analyzed using Recurrent Neural Networks (specifically LSTMs) to process sequences of Mel-scale filterbank spectral features, capturing temporal articulatory changes.
- Fusion: The final prediction for a speech sample is a weighted average of the probability scores from both the DNN and RNN across all segments.
Experimental Setup
The system was evaluated using the INTERSPEECH 2016 Native Language Sub-Challenge dataset, which includes 5,132 speakers across 11 native languages: Arabic (ARA), Chinese (CHI), French (FRE), German (GER), Hindi (HIN), Italian (ITA), Japanese (JPN), Korean (KOR), Spanish (SPA), Telugu (TEL), and Turkish (TUR). Each sample is a 45-second recording, often containing background noise and captured with low-quality microphones.
Results
The proposed hybrid system significantly outperformed the standard competition baseline.
As shown in the results table, the baseline SVM model achieved an Unweighted Average Recall (UAR) of 45.1%. The proposed DNN+RNN fusion model increased this to 52.2%. Further improvement was achieved by fusing the neural model with the baseline, reaching a UAR of 55.8% on the development set. Interestingly, while the neural approach was superior overall, the traditional baseline performed better on specific accents like Spanish and Telugu, suggesting that the two methods capture complementary information.
Conclusion
This research demonstrates that accents are multi-scale phenomena. By combining models that specialize in long-term prosody (DNN) and short-term articulation (RNN), identification accuracy can be significantly improved. Future work may involve hierarchical classification to first identify language groups or language-specific tuning to dynamically weight the DNN and RNN components based on the suspected accent.
Summary updated