BA-MoE: Boundary-Aware Mixture-of-Experts Adapter for Code-Switching Speech Recognition

Peikun Chen, Fan Yu, Yuhao Liang, Hongfei Xue, Xucheng Wan, Naijun Zheng, Huan Zhou, Lei XieASRU 2023

TL;DR

This paper introduces BA-MoE, a novel approach for code-switching Automatic Speech Recognition (ASR) that tackles the challenges of distinguishing between languages and accurately predicting language boundaries. By employing a Mixture-of-Experts (MoE) architecture with language-specific adapters and a boundary-aware training method, BA-MoE significantly reduces errors in code-switching ASR.

What Problem Does This Paper Address?

Code-switching, the practice of alternating between two or more languages within a single utterance, is increasingly common but poses a significant challenge for ASR systems. The core difficulties lie in two areas:

  1. Modeling Language-Specific Representations: Effectively capturing the nuances of each language while also understanding their shared linguistic properties is complex. Similar pronunciations across languages can further complicate this.
  2. Predicting Language Boundaries: Accurately identifying where one language ends and another begins is crucial. Misidentifying these boundaries can lead the ASR model astray, reducing overall accuracy.

Existing methods often struggle with one or both of these issues. Some MoE approaches separate language experts too much, losing valuable cross-lingual interaction. Others implicitly learn boundary information, which can be ambiguous. This paper aims to address these limitations by proposing a more integrated and explicit approach.

Methods

The proposed Boundary-Aware Mixture-of-Experts (BA-MoE) model introduces several key innovations:

  • MoE-Adapter: Instead of completely separate language experts, BA-MoE uses "adapters" within each encoder layer. These adapters extract language-specific representations, which are then fused by a unified gating network. This allows for finer-grained learning of both common and specific representations within each layer, promoting better interaction.
  • Cross-Layer Language Adaptation (CLA) Training: To enhance the adapters' ability to learn language-specific features, a CLA training method is employed. This method uses a cross-layer loss to improve the differentiation between adapters for different languages, leading to more robust language-specific representations.
  • Boundary-Aware Training (BAT): To explicitly address language boundary confusion, BA-MoE incorporates a boundary-aware predictor. This predictor uses self-attention pooling on encoder outputs to generate segment-level boundary representations. These representations are then used to predict language switches, complementing implicit boundary detection. The model is trained to predict both acoustic labels and boundary tokens.

Datasets

The experiments were conducted on the ASRU 2019 Mandarin-English code-switching challenge dataset. This dataset includes approximately 200 hours of code-switching training data and 500 hours of monolingual Mandarin training data. The development and test sets each consist of 20 hours of code-switching data. For experimental design, the authors also included a 460-hour subset of the Librispeech English dataset, similar to previous work.

Key Insights and Ideas

  • Integrated Language Modeling: The MoE-Adapter approach is more effective than strictly separated language experts because it allows for interaction and fusion of representations at each layer, rather than just at the end of the encoder. This is crucial for handling the fluid nature of code-switching.
  • Explicit Boundary Prediction Matters: Simply relying on implicit cues for language boundaries is insufficient. The explicit boundary-aware predictor, leveraging self-attention, significantly improves the model's ability to pinpoint language transitions.
  • Cross-Layer Adaptation Boosts Performance: The CLA training method demonstrates that improving the learning capacity of individual language adapters directly translates to better overall code-switching ASR performance.

Key Results

The BA-MoE model achieved state-of-the-art results on the ASRU 2019 Mandarin-English code-switching challenge dataset:

  • A relative decrease of 16.55% in the mixture error rate (MER) compared to the baseline.
  • A relative decrease of 40.81% in the boundary-switching error rate (BER).
  • The model achieved the lowest CER (10.28%), WER (8.16%), and MER (27.48%) among methods with comparable parameter budgets on the test set.

Limitations and Future Work

  • Parameter Budget: The paper emphasizes comparing models with similar parameter budgets to ensure fair evaluation. While BA-MoE performs well, further exploration with larger models could reveal even greater potential.
  • Dataset Specificity: The current evaluation is on a specific Mandarin-English dataset. Future work could explore the generalizability of BA-MoE to other language pairs and code-switching scenarios.
  • Future Directions: The authors suggest that their approach provides a strong foundation for future research in code-switching ASR, particularly in exploring more sophisticated adapter designs and boundary prediction mechanisms. The effectiveness of the BAT method also opens avenues for research in other sequence-to-sequence tasks with boundary-sensitive elements.

Summary updated

BibTeX
@inproceedings{chen2023ba,
  author = {Peikun Chen and Fan Yu and Yuhao Liang and Hongfei Xue and Xucheng Wan and Naijun Zheng and Huan Zhou and Lei Xie},
  title = {{BA-MoE: Boundary-Aware Mixture-of-Experts Adapter for Code-Switching Speech Recognition}},
  booktitle = {ASRU},
  year = {2023},
}
Figure