A Configurable Multilingual Model is All You Need to Recognize All Languages

Long Zhou, Jinyu Li, Eric Sun, Shujie Liu

TL;DR

Researchers at Microsoft have developed a Configurable Multilingual Model (CMM) that allows users to pre-select any combination of languages they speak. Unlike traditional models that either guess the language or force a single choice, CMM uses a parameter-efficient "universal + residue" architecture to deliver significant accuracy gains (up to 26% error reduction) while remaining lightweight enough for real-world deployment.

What Problem Does This Paper Address?

In our increasingly globalized world, many people are multilingual. However, Automatic Speech Recognition (ASR) systems struggle to keep up. Current industry standards usually take one of two paths:

  1. Universal Models: One giant model trained on all languages. It’s easy to deploy but doesn't perform as well because it lacks specific language "hints."
  2. Language ID (LID) Models: These require the user to pick one language or use a detector to guess. Guessing adds latency, and picking one fails if the user switches languages mid-sentence (code-switching).

The "brute force" solution—building a specific model for every possible combination of languages (e.g., an English-Spanish model, an English-French-German model)—is a scaling nightmare. For just 10 languages, you’d need to build and maintain 120 different trilingual models. The authors sought a way to train one model that can be "configured" on the fly to match any user's specific linguistic profile.

Methods

The CMM is built on a Transformer-Transducer (T-T) backbone. The "secret sauce" lies in how it handles language-specific information:

  • Universal + Residue Architecture: The model consists of a large "universal" module that captures patterns shared across all languages. It then adds very small, language-specific modules (linear layers) that only model the "residue"—the unique nuances of a specific language.
  • Multi-hot Input: Instead of a 1-hot vector (which says "I am speaking English"), the model accepts a multi-hot vector (which says "I might speak English, Spanish, or French"). This vector acts as a toggle, activating the relevant language-specific modules.
  • Language-Specific Vocabulary: At inference time, the model dynamically narrows its "dictionary" to only include tokens from the selected languages. This prevents the system from accidentally outputting a German word when the user has only selected English and Spanish.
  • Training Strategy: To make the model robust, the researchers simulated user choices during training by randomly activating different language combinations for each audio sample.

Datasets

The model was trained on a massive, industrial-scale dataset:

  • 75,000 hours of anonymized Microsoft multilingual data.
  • 10 Languages: English (EN), Spanish (ES), French (FR), Italian (IT), Polish (PL), Portuguese (PT), Dutch (NL), German (DE), Romanian (RO), and Greek (EL).
  • Evaluation: Tested on 10-language test sets and specific code-switching sets (DE/EN and ES/EN) where speakers flip between languages in a single utterance.

Key Insights and Ideas

  • The Power of the Residue: Because most human languages share underlying phonetic structures, the language-specific modules can be tiny. The CMM only adds about 13% more parameters than a standard universal model, making it much more efficient than "Mixture of Experts" (MoE) approaches.
  • The "Confusion" Trade-off: The results show a clear trend: the more languages a user selects, the higher the Word Error Rate (WER). This makes intuitive sense—giving the model more options increases the "confusion" or search space. However, CMM still outperforms universal models across the board.
  • Vocabulary Filtering is Key: One of the simplest but most effective tricks was the "temporary vocabulary" strategy. By simply blocking the model from "seeing" unselected languages during decoding, they significantly reduced errors.

Key Results

The CMM showed impressive improvements over a standard universal multilingual baseline:

  • Single Language Selected: 26.0% relative Word Error Reduction (WERR).
  • Two Languages Selected: 16.9% WERR.
  • Three Languages Selected: 10.4% WERR.
  • Code-Switching: On the Spanish/English code-switching set, the CMM achieved a 16.3% WERR over the baseline, proving it can handle fluid language transitions better than models that expect a single input.

Limitations and Future Work

While the paper demonstrates a major step forward, there are a few areas for further exploration:

  • Scaling to 100+ Languages: The study focused on 10 languages. As the number of languages grows, the "residue" modules might need to become more complex, or the universal module might struggle to hold all that shared information.
  • User Burden: The system relies on the user manually selecting their languages in an interface. Future iterations might look at "soft" selection or automatically updating the multi-hot vector based on the user's history.
  • Parameter Growth: While 13% is small, it is still linear growth. For a system supporting hundreds of dialects, even small linear layers could eventually bloat the model size.

Summary updated

Abstract

Multilingual automatic speech recognition models have shown great promise in recent years because of the simple model training and deployment process. Conventional methods either train a universal multilingual model without taking any language information or with a 1-hot language ID (LID) vector to guide the recognition of the target language. In practice, a multilingual user can be prompted to preselect several languages he/she can speak. The multilingual model without LID cannot well utilize the language information set by the user while the multilingual model with 1-hot LID can only handle one pre-selected language. In this paper, we propose a novel configurable multilingual model (CMM) which is trained only once but can be configured as different models based on users’ choices by extracting language-specific modules together with a universal module from the trained CMM. Particularly, a single CMM can be deployed to any user scenario where the users can pre-select any combination of languages. Trained with 75K hours of transcribed anonymized Microsoft multilingual data and evaluated with 10-language test sets, the proposed CMM improves from the universal multilingual model by 26.0%, 16.9%, and 10.4% relative word error reduction when the user selects 1, 2, or 3 languages, respectively.
BibTeX
@inproceedings{zhou2022configurable,
  author = {Long Zhou and Jinyu Li and Eric Sun and Shujie Liu},
  title = {{A Configurable Multilingual Model is All You Need to Recognize All Languages}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=9OadHR7U4cAM},
}
Figure