TL;DR
Researchers have introduced OMNILINGUAL-MSASR, a new benchmark designed to push "omni-modal" Large Language Models (LLMs) to their limits in transcribing long-form, multi-speaker videos across six languages. The study reveals a significant "temporal gap": while modern models are excellent at transcribing what was said, they struggle immensely with when it was said and who said it, with error rates often exceeding 100% on time-sensitive metrics.
What Problem Does This Paper Address?
Multi-speaker Automatic Speech Recognition (MsASR) is the task of determining who spoke, when they spoke, and what they said. While LLMs have become proficient at standard ASR, real-world scenarios—like meetings, podcasts, or news broadcasts—are far more complex.
Existing benchmarks for MsASR are often limited to short audio clips, focused solely on English or Chinese, and ignore the visual modality. As "omni" models (like GPT-4o or Gemini) emerge with the ability to "see" and "hear" simultaneously, we lack a rigorous way to test if they can actually use visual cues (like lip movements or facial identity) to disambiguate speakers in long, multilingual contexts. OMNILINGUAL-MSASR fills this gap by providing a challenging, multimodal, and long-form evaluation set.
Methods
The authors developed a sophisticated hybrid pipeline to create high-quality ground truth labels for long videos (up to 30 minutes) that exceed the context windows of many specialized tools.
- Audio-Centric Labeling: They used speech enhancement (MossFormer2_SE) followed by a specialized diarization model (MOSS Transcribe Diarize 0.9B) to generate initial "pseudo-labels" for 30-minute clips.
- Visual-Aided Identity Merging: To ensure a speaker's ID remains consistent across different clips of the same video, they employed SyncNet for lip-sync verification and prompted Gemini 3.1 Pro to match faces across segments.
- Human-in-the-Loop Refinement: Finally, linguistic experts conducted a three-stage verification process (Annotation, Review, and Spot-checking) to correct timestamps, speaker labels, and transcripts.
Datasets
The OMNILINGUAL-MSASR benchmark consists of 297 instances sourced from YouTube, covering diverse topics like meetings, TV dramas, and vlogs.
- Languages: Spanish, French, Japanese, Korean, Russian, and Thai.
- Scale: Up to 20 speakers per video.
- Duration: Stratified from short 20-second clips to 30-minute long-form videos.
Key Insights and Ideas
- The Temporal Bottleneck: Models are surprisingly good at linguistic transcription (low
cpWER) but fail at precise temporal grounding. ThetcpWER(time-constrained Word Error Rate) frequently exceeds 100%, meaning models often get the words right but the timestamps completely wrong. - The Visual Modality Paradox: You might expect video to always help, but it doesn't. For models like Doubao-Seed-2.0-Lite, visual cues significantly improved temporal localization. However, for Gemini 3.7 Flash, video actually acted as "confounding noise," degrading its temporal grounding performance.
- Compounding Errors: As video length increases, timestamp errors compound. Similarly, as the number of speakers grows, "speaker confusion" rates spike—rising from 9.1% for two speakers to 25.6% for seven or more.
- Acoustic Challenges: Thai proved to be the most difficult language across all models, largely due to high timbre similarity among speakers, which complicates identity tracking.
Key Results
The researchers evaluated several frontier models, including the Gemini family, Qwen series, and Doubao.
- Instruction Following: Most commercial models followed the complex XML-like output format well, though some (like Gemini 3.7 Flash on French/Korean) saw parse rates drop below 80%.
- Diarization: The Qwen series significantly outperformed others on Diarization Error Rate (DER), showing a clear advantage in handling speech duration.
- Overall Performance: Specialized models like MOSS-Transcribe-Diarize-0.9B still hold a lead over general-purpose omni LLMs in most languages, suggesting that "generalists" haven't yet mastered the nuances of multi-speaker dynamics.
Limitations and Future Work
The authors acknowledge a few key limitations:
- Labeling Bias: Because the ground truth was partially generated using MOSS and Gemini, the benchmark might inherently favor those models.
- Expert Verification Limits: Human experts only corrected "obvious" timestamp errors, meaning some fine-grained noise might remain in the ground truth.
- Stochasticity: The use of commercial APIs means results aren't perfectly reproducible due to the non-deterministic nature of these models.
Future Work: The findings suggest that future omni-modal models need to focus on fine-grained temporal localization and better cross-modal synergy to truly master real-world video transcription.
Summary updated