Scaling A Simple Approach to Zero-Shot Speech Recognition

Jinming Zhao, Vineel Pratap, Michael Auli

TL;DR

Meta FAIR researchers have developed MMS Zero-shot, a simplified approach to speech recognition for thousands of languages that requires no labeled audio for new target languages. By replacing complex phoneme-based systems with a universal romanization strategy and training on over 1,000 languages, they achieved a 46% relative reduction in error rates compared to previous state-of-the-art zero-shot models.

What Problem Does This Paper Address?

Building Automatic Speech Recognition (ASR) systems for the world’s 7,000+ languages is a massive scaling challenge. Traditional supervised learning requires thousands of hours of labeled audio, which only exists for a tiny fraction of languages. Even "unsupervised" methods often require large amounts of both unlabeled audio and text, which are still hard to find for many low-resource tongues.

Recent "zero-shot" attempts tried to solve this by predicting language-independent "allophones" (speech sounds) and mapping them to phonemes. However, this relies on a "phonemizer"—a tool that converts text to sounds—which is often inaccurate or unavailable for obscure languages. The authors identify this phoneme bottleneck as the primary hurdle to truly universal ASR.

Methods

The core innovation of MMS Zero-shot is its simplicity: it ditches phonemes entirely in favor of romanization.

  1. Universal Romanization: The researchers use a tool called uroman to convert text from any script (Cyrillic, Arabic, Devanagari, etc.) into a standardized Latin-based representation. This creates a shared "alphabet" across all languages.
  2. Universal Acoustic Model: They fine-tune a massive pre-trained self-supervised model (XLS-R, based on wav2vec 2.0) on labeled data from 1,078 languages. Instead of predicting characters or phonemes, the model is trained to output these universal romanized tokens.
  3. Zero-Shot Inference: To support a new language, you only need a list of words (a lexicon) in that language. You romanize that list, and during decoding, the model uses beam search to find the most likely sequence of words from your lexicon that matches the romanized acoustic output.
The MMS Zero-shot training and inference pipeline.

Datasets

The study leverages several massive multilingual datasets:

  • MMS-lab: Based on New Testament recordings, providing paired data for 1,107 languages.
  • FLEURS: A high-quality ASR benchmark covering 102 languages.
  • CommonVoice (CV): A crowdsourced multilingual dataset.
  • Crúbadán & Panlex: External text databases used to build lexicons for thousands of low-resource languages.

The final model was trained on approximately 39,946 hours of data across 1,078 languages.

Key Insights and Ideas

  • Romanization > Phonemes: Standardizing text to a Latin script is more robust than trying to guess exact pronunciations (phonemes) for unseen languages. Phonemizers often fail on rare languages because they lack linguistic rules; romanization is a simpler, more reliable heuristic.
  • The Power of Scale: By training on three orders of magnitude more languages than prior work (1,078 vs. 8), the acoustic model learns a "universal" representation of speech that generalizes remarkably well to languages it has never heard.
  • Text is the Key: The accuracy of zero-shot ASR is highly sensitive to the quality and amount of available text data for the target language. Even a few thousand sentences of unlabeled text can significantly boost performance by allowing for better lexicons and unigram language models.
Impact of increasing text data on Character Error Rate (CER).

Key Results

The results demonstrate a significant leap in zero-shot capabilities:

  • 46% Relative Improvement: MMS Zero-shot reduced the average Character Error Rate (CER) across 100 unseen languages by 46% compared to the previous best model, ASR-2K.
  • Closing the Gap: The error rate of this zero-shot approach is only 2.5x higher than supervised baselines that were trained specifically on the target languages. For a system that has never seen a single labeled example of the target language, this is a remarkably narrow gap.
  • In-Domain vs. Out-of-Domain: The model performed exceptionally well on MMS-lab (biblical text) but saw higher error rates on FLEURS, suggesting that the "domain" of the training text still matters.
Comparison of MMS Zero-shot against the ASR-2K baseline across multiple datasets.

Limitations and Future Work

  • Domain Sensitivity: Because the bulk of the training data (MMS-lab) is religious text, the model struggles with more diverse or modern vocabulary found in datasets like FLEURS.
  • Romanization Ambiguity: Romanization can be "many-to-one" (different characters mapping to the same Latin letter), which introduces ambiguity that the model must resolve during decoding.
  • Data Scarcity for Lexicons: While the model doesn't need audio for new languages, it still needs some text to build a lexicon. For the most endangered languages, even finding a clean word list remains a challenge.

Future work will likely focus on expanding the diversity of the training text and exploring more sophisticated language modeling to further bridge the gap between zero-shot and supervised performance.

Summary updated

Abstract

Despite rapid progress in increasing the language coverage of automatic speech recognition, the field is still far from covering all languages with a known writing script. Recent work showed promising results with a zero-shot approach requiring only a small amount of text data, however, accuracy heavily depends on the quality of the used phonemizer which is often weak for unseen languages. In this paper, we present MMS Zero-shot, a conceptually simpler approach based on romanization and an acoustic model trained on data in 1,078 different languages or three orders of magnitude more than prior art. MMS Zero-shot reduces the average character error rate by a relative 46% over 100 unseen languages compared to the best previous work. Moreover, the error rate of our approach is only 2.5x higher than in-domain supervised baselines, while MMS Zero-shot uses no labeled data for the evaluation languages at all.
BibTeX
@inproceedings{zhao2025scaling,
  author = {Jinming Zhao and Vineel Pratap and Michael Auli},
  title = {{Scaling A Simple Approach to Zero-Shot Speech Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=eaaQzMvt10},
}

Loading related content...

Figure