LAMA-UT: Language Agnostic Multilingual ASR Through Orthography Unification and Language-Specific Transliteration

Sangmin Lee, Woo-Jin Chung, Hong-Goo KangAAAI 2025 · DOI

TL;DR

This paper introduces LAMA-UT, a novel multilingual Automatic Speech Recognition (ASR) pipeline that achieves state-of-the-art performance across over 100 languages, including unseen ones, without relying on any language-specific modules. By unifying orthographic systems through Romanization and using a large language model (LLM) as a universal converter, LAMA-UT demonstrates remarkable efficiency and generalization, even with minimal training data.

What Problem Does This Paper Address?

Building a single ASR model that performs well across a vast number of languages, especially low-resource or entirely new ones, is a significant challenge. Existing approaches either scale up models and datasets to cover many languages, which can lead to inconsistent performance, or they incorporate language-specific modules (like adapters or LMs) into a universal model. While the latter improves performance, it compromises the ideal of a truly universal, language-agnostic system and can complicate the pipeline. This paper aims to bridge this gap by proposing a method that achieves high, consistent multilingual ASR performance without any language-specific components, making it more flexible and generalizable.

Methods

LAMA-UT operates in two main phases:

  1. Universal Transcription Generation: This phase focuses on unifying diverse orthographic systems into a single, consistent format. Instead of using the International Phonetic Alphabet (IPA), which can be complex and require specific rules for each language, the authors opt for Romanization. This method converts text from various writing systems into the Latin alphabet, which is more efficient and aligns better with the training data of large language models. The goal is to capture phonetic features in a standardized way, creating a "universal transcription." This is achieved by fine-tuning a self-supervised learning (SSL) model (XLS-R) with a classification layer on speech and Romanized transcription pairs.
  2. Language-Specific Transliteration: In this phase, the universal, Romanized transcriptions are converted back into language-specific ones. This is framed as a text-to-text transliteration task, leveraging the power of large language models (LLMs) as universal converters. The LLM is prompted to perform this conversion without any fine-tuning, making the converter interchangeable with any superior LLM. Various prompting strategies (zero-shot, few-shot, CoT, prompt chaining) were explored to find the most effective way to guide the LLM.

Datasets

  • FLEURS: Used for training and evaluating the pipeline on over 100 languages, noted for its relatively small data per language.
  • Common Voice 17.0: Used for evaluating performance on unseen languages, specifically the test split of 25 languages.

Key Insights and Ideas

  • Romanization over IPA for Unification: The paper argues that Romanization is a more practical and effective method for orthographic unification than IPA. It simplifies the process, aligns better with LLM training data, and still captures sufficient phonetic information for multilingual ASR.
  • LLMs as Universal Converters: Demonstrating that a frozen LLM can effectively act as a universal converter for transliterating Romanized transcriptions into language-specific ones is a key insight. This highlights the emergent capabilities of LLMs for cross-lingual tasks without task-specific fine-tuning.
  • Minimal Data, Maximum Impact: The pipeline achieves impressive results with a fraction of the data used by state-of-the-art models like Whisper and MMS, underscoring the efficiency of the proposed approach, especially for low-resource languages.
  • Zero-Shot Generalization: The ability to perform comparably to models that do use language-specific modules, even when LAMA-UT does not, is a significant achievement, particularly for unseen languages.

Key Results

  • Seen Languages: LAMA-UT achieved a relative reduction of 60% in CER and 30% in WER compared to Whisper. It matched the performance of MMS despite using only 0.1% of Whisper's training data and without language-specific modules.
  • Unseen Languages: LAMA-UT reduced CER by half compared to previous zero-shot ASR models (ASR-2K) while using significantly less training data. Its performance was comparable to MMS-ZS, which relies on language-specific lexicons and LMs.
  • Prompting Strategy: Few-shot prompting generally yielded the best results, with GPT-40-mini achieving an impressive average CER of 15% across 102 languages.
  • Universal Converter Performance: The "upper bound" performance of the universal converter (using ground truth Romanized transcriptions) showed strong results, with GPT-40-mini achieving a CER of 7.5% and WER of 18.1% on seen languages.

Limitations and Future Work

  • Prompting Strategy Effectiveness: While few-shot prompting showed the best results, sequential reasoning strategies like Zero-Shot CoT and prompt chaining did not yield anticipated improvements, sometimes leading to error propagation.
  • LLM Issues: Smaller LLMs like LLaMA-8B exhibited issues with word repetition and language misprediction, particularly for low-resource languages. GPT-40-mini performed best overall.
  • Unseen Language Performance Dip: The performance on unseen languages showed a slight decrease compared to seen languages, attributed to the LLM's training data potentially having fewer examples of these extremely low-resource languages.
  • Future Work: The authors anticipate that this framework can serve as a cornerstone for flexible multilingual ASR systems, generalizable even to unseen languages. They suggest that advancements in LLMs could further improve the universal converter's performance.

Summary updated

BibTeX
@inproceedings{lee2025lama,
  author = {Sangmin Lee and Woo-Jin Chung and Hong-Goo Kang},
  title = {{LAMA-UT: Language Agnostic Multilingual ASR Through Orthography Unification and Language-Specific Transliteration}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i23.34617},
}
Figure