Omnilingual-MsASR: Benchmarking Multimodal, Multilingual, Long-Form Multi-Speaker ASR of Omni Large Language Models

Yuan Ge, Pei Zhang, Xize Cheng, Lingzhou Mu, Mingyang Wang, Jin Xu, Baosong Yang, Shengxiang Gao, Jingbo Zhu, Zhengtao Yu, Tong XiaoarXiv 2026 · arXiv

TL;DR

Researchers have introduced OMNILINGUAL-MSASR, a new benchmark designed to push "omni-modal" Large Language Models (LLMs) to their limits in transcribing long-form, multi-speaker videos across six languages. The study reveals a significant "temporal gap": while modern models are excellent at transcribing what was said, they struggle immensely with when it was said and who said it, with error rates often exceeding 100% on time-sensitive metrics.

What Problem Does This Paper Address?

Multi-speaker Automatic Speech Recognition (MsASR) is the task of determining who spoke, when they spoke, and what they said. While LLMs have become proficient at standard ASR, real-world scenarios—like meetings, podcasts, or news broadcasts—are far more complex.

Existing benchmarks for MsASR are often limited to short audio clips, focused solely on English or Chinese, and ignore the visual modality. As "omni" models (like GPT-4o or Gemini) emerge with the ability to "see" and "hear" simultaneously, we lack a rigorous way to test if they can actually use visual cues (like lip movements or facial identity) to disambiguate speakers in long, multilingual contexts. OMNILINGUAL-MSASR fills this gap by providing a challenging, multimodal, and long-form evaluation set.

Figure 1: Overview of the MsASR task. The model is required to jointly predicting speaker identities (who), timestamps (when), and text transcriptions (what) in a structured format as shown in Sec. 4.2.

Methods

The authors developed a sophisticated hybrid pipeline to create high-quality ground truth labels for long videos (up to 30 minutes) that exceed the context windows of many specialized tools.

  1. Audio-Centric Labeling: They used speech enhancement (MossFormer2_SE) followed by a specialized diarization model (MOSS Transcribe Diarize 0.9B) to generate initial "pseudo-labels" for 30-minute clips.
  2. Visual-Aided Identity Merging: To ensure a speaker's ID remains consistent across different clips of the same video, they employed SyncNet for lip-sync verification and prompted Gemini 3.1 Pro to match faces across segments.
  3. Human-in-the-Loop Refinement: Finally, linguistic experts conducted a three-stage verification process (Annotation, Review, and Spot-checking) to correct timestamps, speaker labels, and transcripts.
The automatic data pipeline combining audio diarization and visual face merging.

Datasets

The OMNILINGUAL-MSASR benchmark consists of 297 instances sourced from YouTube, covering diverse topics like meetings, TV dramas, and vlogs.

  • Languages: Spanish, French, Japanese, Korean, Russian, and Thai.
  • Scale: Up to 20 speakers per video.
  • Duration: Stratified from short 20-second clips to 30-minute long-form videos.
Figure 2: Benchmark composition of OMNILINGUAL-MSASR in three views: languages, audio duration and speaker count. For each view we shows case quantity, category labels, and necessary explanations.

Key Insights and Ideas

  • The Temporal Bottleneck: Models are surprisingly good at linguistic transcription (low cpWER) but fail at precise temporal grounding. The tcpWER (time-constrained Word Error Rate) frequently exceeds 100%, meaning models often get the words right but the timestamps completely wrong.
  • The Visual Modality Paradox: You might expect video to always help, but it doesn't. For models like Doubao-Seed-2.0-Lite, visual cues significantly improved temporal localization. However, for Gemini 3.7 Flash, video actually acted as "confounding noise," degrading its temporal grounding performance.
  • Compounding Errors: As video length increases, timestamp errors compound. Similarly, as the number of speakers grows, "speaker confusion" rates spike—rising from 9.1% for two speakers to 25.6% for seven or more.
  • Acoustic Challenges: Thai proved to be the most difficult language across all models, largely due to high timbre similarity among speakers, which complicates identity tracking.

Key Results

The researchers evaluated several frontier models, including the Gemini family, Qwen series, and Doubao.

  • Instruction Following: Most commercial models followed the complex XML-like output format well, though some (like Gemini 3.7 Flash on French/Korean) saw parse rates drop below 80%.
  • Diarization: The Qwen series significantly outperformed others on Diarization Error Rate (DER), showing a clear advantage in handling speech duration.
  • Overall Performance: Specialized models like MOSS-Transcribe-Diarize-0.9B still hold a lead over general-purpose omni LLMs in most languages, suggesting that "generalists" haven't yet mastered the nuances of multi-speaker dynamics.

Limitations and Future Work

The authors acknowledge a few key limitations:

  • Labeling Bias: Because the ground truth was partially generated using MOSS and Gemini, the benchmark might inherently favor those models.
  • Expert Verification Limits: Human experts only corrected "obvious" timestamp errors, meaning some fine-grained noise might remain in the ground truth.
  • Stochasticity: The use of commercial APIs means results aren't perfectly reproducible due to the non-deterministic nature of these models.

Future Work: The findings suggest that future omni-modal models need to focus on fine-grained temporal localization and better cross-modal synergy to truly master real-world video transcription.

Summary updated

Abstract

Omni-modal large language models (LLMs) have achieved remarkable progress in audio and video understanding. However, existing models struggle with long-form, multi-speaker videos, especially in understanding the entire video and recognizing who says what and when. To characterize the capability limit of current omni LLMs and guide future model development, we introduce Omnilingual-MsASR, a multilingual, multimodal benchmark for long-form Multi-speaker Automatic Speech Recognition (MsASR) covering six languages, recordings up to 30 minutes, and scenarios with up to 20 speakers. Omnilingual-MsASR provides parallel audio and video inputs to evaluate models' cross-modal consistency and their ability to exploit complementary cues across modalities. Extensive evaluations of leading omni-modal LLMs reveal substantial performance bottlenecks, while their tcpWERs frequently exceed 100%. These findings highlight fine-grained temporal localization, speaker disambiguation, and cross-modal synergy as key challenges for future omni-modal foundation models. Our work is available at https://github.com/QwenLM/Omnilingua-Bench/tree/main/Omnilingua-MSpeaker.
BibTeX
@misc{ge2026omnilingual,
  author = {Yuan Ge and Pei Zhang and Xize Cheng and Lingzhou Mu and Mingyang Wang and Jin Xu and Baosong Yang and Shengxiang Gao and Jingbo Zhu and Zhengtao Yu and Tong Xiao},
  title = {{Omnilingual-MsASR: Benchmarking Multimodal, Multilingual, Long-Form Multi-Speaker ASR of Omni Large Language Models}},
  year = {2026},
  eprint = {2610.29455},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.29455},
}
Figure