Abstract
Multi-task learning (MTL) in deep neural networks has been proven to be effective for acoustic modeling in speech recognition. In this paper, this technique is applied to Mandarin-English code-mixing recognition. For the primary task of senone classification, three schemes of auxiliary tasks are proposed to introduce language information to the networks and improve the prediction of language switching. On a real-world Mandarin-English test corpus from mobile voice search, the proposed schemes enhanced recognition in both languages and reduced the relative overall error rates by 3.5%, 3.8%, and 5.8%, respectively.
Introduction
In many bilingual communities, "code-mixing"—switching between languages like Mandarin and English within a single sentence—is a standard part of daily conversation. However, for Automatic Speech Recognition (ASR) systems, this phenomenon presents a significant challenge. The primary issue is data sparsity, as there is often insufficient recorded audio of language switching to train robust models.
Furthermore, standard ASR systems utilize "triphone state clustering" to group similar sounds (senones). In code-mixing environments, this process frequently merges bilingual sounds with monolingual ones. For instance, an English phoneme preceded by a Mandarin one might be clustered with a purely English sound, effectively "erasing" the signal that a language switch occurred. This makes it difficult for systems to accurately predict switch points.
Methodology: Multi-Task Learning for Code-Mixing
To address these challenges, the authors propose using Multi-Task Learning in Deep Neural Networks (MTL-DNN). In this architecture, the lower hidden layers of the network are shared across multiple tasks. While the primary task remains senone classification, the network is simultaneously trained on auxiliary tasks that provide additional language structure information.
Three specific auxiliary schemes were evaluated:
- Phoneme Language Task (phnLan): The network predicts if a sound is Mandarin, English, or non-speech, forcing the hidden layers to learn fundamental language differences.
- Phoneme Task: The network predicts the central phoneme, helping the model learn similarities between senones sharing the same root sound.
- Combined Task: The network predicts the phoneme and the language of the neighboring phonemes. This approach recovers language context information typically lost during clustering.
During inference, the auxiliary output layers are discarded, ensuring the final model size remains identical to a standard DNN while benefiting from the "smarter" internal weights developed during training.
Experimental Setup
The models were trained on a 236-hour acoustic dataset comprising 100 hours of Mandarin, 100 hours of English, and 36 hours of in-domain Mandarin-English code-mixed data. Evaluation was performed on a real-world test corpus of 1,200 Mandarin-English utterances from a mobile voice search application, where approximately 41% of sentences contained at least one English word.
Results and Analysis
The researchers compared the MTL-DNN schemes against a standard single-task DNN baseline using the Mixed Error Rate (MER). The results demonstrate that all MTL schemes outperformed the baseline, with the Combined Task providing the most significant improvement.
Key findings include:
- The Combined Task Advantage: By providing information about the language of the surrounding context, the Combined Task reduced the MER from 34.2% to 32.2%, a 5.8% relative reduction.
- Asymmetric Improvement: MTL improved English recognition more significantly than Mandarin. The authors suggest English accuracy is more sensitive to switch detection, as English words misrecognized as Mandarin characters lead to cascading errors.
- Language Context as a Regularizer: Injecting structural language knowledge via auxiliary tasks acts as an effective regularizer without increasing the final model's computational footprint.
Conclusion
Multi-task learning provides a robust framework for improving code-mixed speech recognition by addressing the limitations of traditional triphone clustering. Future work may explore dynamic weighting schedules for auxiliary tasks and the application of these techniques to modern architectures like LSTMs and Transformers, as well as other language pairs.
Summary updated