Abstract
Recently, to mitigate the confusion between different languages in code-switching (CS) automatic speech recognition (ASR), the conditionally factorized models, such as the language-aware encoder (LAE), explicitly disregard the contextual information between different languages. However, this information may be helpful for ASR modeling. To alleviate this issue, we propose the LAE-ST-MoE framework. It incorporates speech translation (ST) tasks into LAE and utilizes ST to learn the contextual information between different languages. It introduces a task-based mixture of expert modules, employing separate feed-forward networks for the ASR and ST tasks. Experimental results on the ASRU 2019 Mandarin-English CS challenge dataset demonstrate that, compared to the LAE-based CTC, the LAE-ST-MoE model achieves a 9.26% mix error reduction on the CS test with the same decoding parameter. Moreover, the well-trained LAE-ST-MoE model can perform ST tasks from CS speech to Mandarin or English text.
Introduction
Code-switching (CS)—the practice of alternating between two or more languages in a single conversation—is a major hurdle for ASR systems. Most models suffer from "language confusion," where the system struggles to distinguish between phonetically similar sounds from different languages. To solve this, Language-Aware Encoders (LAE) were developed to factorize the problem using separate encoders for each language. However, by strictly separating the languages, LAE models ignore the contextual relationship between them. In a sentence like "真正做到 happy every day," the Mandarin and English parts are semantically linked. Capturing this cross-lingual context is essential for more accurate recognition.
Methodology
The proposed LAE-ST-MoE framework introduces two key innovations to the standard LAE architecture:
- Speech Translation (ST) as an Auxiliary Task: The model is trained to transcribe speech (ASR) and translate it (e.g., CS speech to pure Mandarin or English text). This forces the shared encoder to learn a deeper, semantic representation of how the two languages interact.
- Task-Based Mixture of Experts (MoE): To prevent "task interference" between ASR and ST, the authors replace standard Feed-Forward Networks (FFNs) with an MoE module. This module uses separate "experts" (FFNs) for the ASR and ST tasks, allowing the model to specialize its internal representations for each goal while still sharing low-level acoustic information.
During ASR inference, the ST decoders and translation-specific experts are discarded, ensuring zero additional decoding latency compared to the baseline.
Experiments
The model was evaluated using the ASRU 2019 Mandarin-English CS Challenge dataset, along with Librispeech (English) and ASRU-Man (Mandarin) for monolingual data. Since the ASRU dataset lacks translation transcripts, pseudo-labels for the ST auxiliary task were generated using a pre-trained Machine Translation (MT) model based on the CSANMT algorithm.
Results
The LAE-ST-MoE model was tested against several baselines, including "Vanilla" CTC models and the standard LAE.
Key findings include:
- 9.26% Relative Error Reduction: On the ASRU 2019 code-switching test set, the proposed model (S3) achieved a significant reduction in Mix Error Rate (MER) compared to the LAE baseline (S2).
- Monolingual Improvements: The benefits extended to pure Mandarin and English speech, with relative error reductions of 7.55% and 8.57%, respectively.
- Attention-based Gains: Similar improvements were observed in attention-based systems (S6 vs S5), with a 7% relative reduction on the CS test set.
Conclusion
The LAE-ST-MoE framework demonstrates that embracing the relationship between languages via translation auxiliary tasks makes the encoder more robust. The task-based MoE effectively manages multi-task learning without performance degradation. Future work will explore one-to-many ST tasks and more complex multilingual scenarios.
Summary updated