mSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Tasks

Luel Hagos Beyene, Vivek Verma, Min Ma, Jesujoba O. Alabi, Fabian David Schmidt, Joyce Nakatumba-Nabende, David Ifeoluwa AdelaniarXiv 2025 · DOI · arXiv

The rapid advancement of Large Language Models (LLMs) has brought about impressive multimodal and multilingual capabilities. However, evaluation frameworks have largely failed to keep pace, remaining heavily skewed toward English and a select group of high-resource languages. While text-based benchmarks exist for a wider range of languages, speech evaluations are often restricted to basic tasks like Automatic Speech Recognition (ASR) or Language Identification (LID), neglecting complex reasoning tasks like audio-based Question Answering (QA).

To address this, researchers introduced mSTEB, a massively multilingual benchmark designed to evaluate LLMs across both speech and text modalities in over 200 languages. This study provides a standardized measure of how state-of-the-art models perform on low-resource languages across a diverse set of tasks.

The mSTEB Benchmark and Methodology

mSTEB aggregates several existing high-quality datasets and introduces a new contribution to the field: Global NLI. This new text-based Natural Language Inference dataset was curated by aggregating 12 human-curated NLI datasets (including XNLI and AfriXNLI), covering 59 languages with 600 samples per language.

The benchmark spans five tasks per modality:

  • Text: LID, Topic Classification, Reading Comprehension QA, Machine Translation, and Natural Language Inference (NLI).
  • Speech: LID, Topic Classification, Reading Comprehension QA, Speech-to-Text Translation, and ASR.

The researchers conducted zero-shot evaluations of five prominent models:

  • Proprietary: GPT-4o (Text/Audio) and Gemini 2.0 Flash (Text/Audio).
  • Open-weight: Qwen 2 Audio (Speech) and Gemma 3 27B (Text).

To provide context, these models were compared against task-specific baselines such as MMS-LID, SeamlessM4T v2, and NLLB. All LLMs were prompted in English to ensure optimal performance across multilingual tasks.

Key Findings: Generalists vs. Specialists

One of the most significant insights from the study is that generalist multimodal LLMs are beginning to outperform specialized models. Gemini 2.0 Flash, for instance, surpassed SeamlessM4T v2—a dedicated speech translation model—on complex tasks like ASR and Speech-to-Text Translation. This suggests a paradigm shift where massive scale and multimodal training allow generalists to handle low-resource languages more effectively than task-specific architectures.

The Modality and Proprietary Gaps

Despite the success of generalists, a clear "modality gap" persists. Models consistently perform better on text-based tasks than on the equivalent tasks in the speech modality. This indicates that audio-LLM architectures still require significant refinement to reach the maturity of their text-only counterparts.

Furthermore, the gap between proprietary and open-weight models is starkest in the audio domain. While Gemma 3 27B (open-weight) proved highly competitive with GPT-4o in text tasks for European and Asian languages, the open-weight audio model, Qwen 2 Audio, struggled significantly. It failed entirely on many low-resource languages, scoring 0.0% accuracy on African LID tasks.

Geographic Disparities and Error Analysis

The evaluation revealed a severe geographic disparity in model performance. While Gemini 2.0 Flash maintained a Character Error Rate (CER) under 15 for ASR in most regions, performance plummeted for African languages (18.0 CER for Gemini, and a staggering 60.1 for GPT-4o).

In the text domain, the Global NLI benchmark highlighted similar issues. Languages from the Americas and Oceania saw an average accuracy of only 49.1%, compared to 84.8% for Western, Northern, and Southern European languages.

Interestingly, the study found that when audio models fail at LID, they tend to confuse languages within the same geographic region. This suggests that models are capturing shared phonetic or linguistic traits but lack the granularity to distinguish between specific low-resource languages within a family.

Conclusion

The mSTEB benchmark exposes the "digital divide" in current AI development. While proprietary models are making strides in multimodal reasoning, the performance drop-off for African, American, and Oceanian languages remains a critical hurdle. The authors argue that the research community must prioritize data collection and model development for these under-represented regions to ensure that the benefits of multimodal LLMs are globally accessible.

Summary updated

Abstract

Large Language models (LLMs) have demonstrated impressive performance on a wide range of tasks, including in multimodal settings such as speech. However, their evaluation is often limited to English and a few high-resource languages. For low-resource languages, there is no standardized evaluation benchmark. In this paper, we address this gap by introducing mSTEB, a new benchmark to evaluate the performance of LLMs on a wide range of tasks covering language identification, text classification, question answering, and translation tasks on both speech and text modalities. We evaluated the performance of leading LLMs such as Gemini 2.0 Flash and GPT-4o (Audio) and state-of-the-art open models such as Qwen 2 Audio and Gemma 3 27B. Our evaluation shows a wide gap in performance between high-resource and low-resource languages, especially for languages spoken in Africa and Americas/Oceania. Our findings show that more investment is needed to address their under-representation in LLMs coverage.
BibTeX
@misc{beyene2025msteb,
  author = {Luel Hagos Beyene and Vivek Verma and Min Ma and Jesujoba O. Alabi and Fabian David Schmidt and Joyce Nakatumba-Nabende and David Ifeoluwa Adelani},
  title = {{mSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Tasks}},
  year = {2025},
  eprint = {2506.08400},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.08400},
}
Figure