The rapid advancement of Large Language Models (LLMs) has brought about impressive multimodal and multilingual capabilities. However, evaluation frameworks have largely failed to keep pace, remaining heavily skewed toward English and a select group of high-resource languages. While text-based benchmarks exist for a wider range of languages, speech evaluations are often restricted to basic tasks like Automatic Speech Recognition (ASR) or Language Identification (LID), neglecting complex reasoning tasks like audio-based Question Answering (QA).
To address this, researchers introduced mSTEB, a massively multilingual benchmark designed to evaluate LLMs across both speech and text modalities in over 200 languages. This study provides a standardized measure of how state-of-the-art models perform on low-resource languages across a diverse set of tasks.
The mSTEB Benchmark and Methodology
mSTEB aggregates several existing high-quality datasets and introduces a new contribution to the field: Global NLI. This new text-based Natural Language Inference dataset was curated by aggregating 12 human-curated NLI datasets (including XNLI and AfriXNLI), covering 59 languages with 600 samples per language.
The benchmark spans five tasks per modality:
- Text: LID, Topic Classification, Reading Comprehension QA, Machine Translation, and Natural Language Inference (NLI).
- Speech: LID, Topic Classification, Reading Comprehension QA, Speech-to-Text Translation, and ASR.
The researchers conducted zero-shot evaluations of five prominent models:
- Proprietary: GPT-4o (Text/Audio) and Gemini 2.0 Flash (Text/Audio).
- Open-weight: Qwen 2 Audio (Speech) and Gemma 3 27B (Text).
To provide context, these models were compared against task-specific baselines such as MMS-LID, SeamlessM4T v2, and NLLB. All LLMs were prompted in English to ensure optimal performance across multilingual tasks.
Key Findings: Generalists vs. Specialists
One of the most significant insights from the study is that generalist multimodal LLMs are beginning to outperform specialized models. Gemini 2.0 Flash, for instance, surpassed SeamlessM4T v2—a dedicated speech translation model—on complex tasks like ASR and Speech-to-Text Translation. This suggests a paradigm shift where massive scale and multimodal training allow generalists to handle low-resource languages more effectively than task-specific architectures.
The Modality and Proprietary Gaps
Despite the success of generalists, a clear "modality gap" persists. Models consistently perform better on text-based tasks than on the equivalent tasks in the speech modality. This indicates that audio-LLM architectures still require significant refinement to reach the maturity of their text-only counterparts.
Furthermore, the gap between proprietary and open-weight models is starkest in the audio domain. While Gemma 3 27B (open-weight) proved highly competitive with GPT-4o in text tasks for European and Asian languages, the open-weight audio model, Qwen 2 Audio, struggled significantly. It failed entirely on many low-resource languages, scoring 0.0% accuracy on African LID tasks.
Geographic Disparities and Error Analysis
The evaluation revealed a severe geographic disparity in model performance. While Gemini 2.0 Flash maintained a Character Error Rate (CER) under 15 for ASR in most regions, performance plummeted for African languages (18.0 CER for Gemini, and a staggering 60.1 for GPT-4o).
In the text domain, the Global NLI benchmark highlighted similar issues. Languages from the Americas and Oceania saw an average accuracy of only 49.1%, compared to 84.8% for Western, Northern, and Southern European languages.
Interestingly, the study found that when audio models fail at LID, they tend to confuse languages within the same geographic region. This suggests that models are capturing shared phonetic or linguistic traits but lack the granularity to distinguish between specific low-resource languages within a family.
Conclusion
The mSTEB benchmark exposes the "digital divide" in current AI development. While proprietary models are making strides in multimodal reasoning, the performance drop-off for African, American, and Oceanian languages remains a critical hurdle. The authors argue that the research community must prioritize data collection and model development for these under-represented regions to ensure that the benefits of multimodal LLMs are globally accessible.
Summary updated