Abstract
Despite rapid progress in increasing the language coverage of automatic speech recognition, the field is still far from covering all languages with a known writing script. Recent work showed promising results with a zero-shot approach requiring only a small amount of text data, however, accuracy heavily depends on the quality of the used phonemizer which is often weak for unseen languages. In this paper, we present MMS Zero-shot a conceptually simpler approach based on romanization and an acoustic model trained on data in 1,078 different languages or three orders of magnitude more than prior art. MMS Zero-shot reduces the average character error rate by a relative 46% over 100 unseen languages compared to the best previous work. Moreover, the error rate of our approach is only 2.5x higher compared to in-domain supervised baselines, while our approach uses no labeled data for the evaluation languages at all.
Introduction
Building Automatic Speech Recognition (ASR) systems for the world’s 7,000+ languages is a massive scaling challenge. Traditional supervised learning requires thousands of hours of labeled audio, which only exists for a tiny fraction of languages. Even "unsupervised" methods often require large amounts of both unlabeled audio and text, which are still hard to find for many low-resource tongues.
Recent "zero-shot" attempts tried to solve this by predicting language-independent "allophones" (speech sounds) and mapping them to phonemes. However, this relies on a "phonemizer"—a tool that converts text to sounds—which is often inaccurate or unavailable for obscure languages. Meta FAIR researchers have developed MMS Zero-shot, a simplified approach that ditches phonemes entirely in favor of universal romanization.
Methods
The core innovation of MMS Zero-shot is replacing complex phoneme-based systems with a universal romanization strategy.
- Universal Romanization: The researchers use
uromanto convert text from any script into a standardized Latin-based representation, creating a shared alphabet across all languages. - Universal Acoustic Model: A massive pre-trained self-supervised model (XLS-R) is fine-tuned on labeled data from 1,078 languages to output these universal romanized tokens.
- Zero-Shot Inference: Supporting a new language requires only a lexicon (word list). This list is romanized, and during decoding, the model uses beam search to find the most likely sequence of words matching the acoustic output.
Datasets
The study leverages several massive multilingual datasets to achieve its scale:
- MMS-lab: Paired data for 1,107 languages based on New Testament recordings.
- FLEURS: A high-quality ASR benchmark covering 102 languages.
- CommonVoice (CV): A crowdsourced multilingual dataset.
- Crúbadán & Panlex: External text databases used to build lexicons for thousands of low-resource languages.
The final model was trained on approximately 39,946 hours of data across 1,078 languages.
Results and Insights
MMS Zero-shot demonstrates a significant leap in zero-shot capabilities, achieving a 46% relative reduction in Character Error Rate (CER) across 100 unseen languages compared to the previous state-of-the-art model, ASR-2K.
Key insights from the evaluation include:
- Romanization > Phonemes: Standardizing text to a Latin script is more robust than guessing pronunciations for unseen languages. Romanization is a simpler, more reliable heuristic than phonemizers, which often fail on rare languages.
- The Power of Scale: Training on 1,078 languages (vs. 8 in prior work) allows the acoustic model to learn a universal representation of speech that generalizes to unheard languages.
- Closing the Gap: The zero-shot error rate is only 2.5x higher than supervised baselines. For a system with no labeled examples of the target language, this is a remarkably narrow gap.
Limitations and Future Work
The model shows sensitivity to domain; performance is strongest on biblical text (MMS-lab) but higher on diverse datasets like FLEURS. Additionally, romanization can be ambiguous (many-to-one mappings), and building lexicons still requires some text data, which remains scarce for the most endangered languages. Future work will focus on expanding training text diversity and improving language modeling to further bridge the gap to supervised performance.
Summary updated