MultiMed-ST: Large-scale Many-to-many Multilingual Medical Speech Translation

Khai Le-Duc, Tuyen Tran, Bach Phan Tat, Nam Q. Bui, Quan Anh, Hung-Phong Tran, Thanh Thuy Nguyen, Thi Ly Nguyen, Thanh Cong Phan, Thi Thu Phuong Tran, Chong‐Wah Ngo, Khanh Nguyen, Thanh Nguyen-Tang2025 · DOI

Introduction

Effective communication is the cornerstone of safe and effective healthcare. When language barriers exist between clinicians and patients, the risk of misdiagnosis and improper treatment increases significantly. While machine translation (MT) and speech translation (ST) have seen rapid advancements, their application in the medical domain remains limited. This is primarily due to a severe lack of high-quality, open-access medical speech data, which is often restricted by privacy regulations like HIPAA.

To address this gap, the authors present MultiMed-ST, the largest many-to-many multilingual medical speech translation dataset to date. This work provides a comprehensive benchmark for the community, evaluating various architectures and training strategies to determine the most effective approach for high-stakes medical communication.

The MultiMed-ST Dataset

MultiMed-ST represents a significant leap in available resources for medical AI. It consists of approximately 290,000 utterances, totaling roughly 150 hours of speech. The dataset covers five languages—Vietnamese, English, German, French, and Chinese (Simplified and Traditional)—encompassing all 20 possible translation directions.

The dataset was constructed using a rigorous pipeline:

  1. Source Data: The process began with a real-world medical Automatic Speech Recognition (ASR) dataset.
  2. Initial Translation: The Gemini Large Language Model (LLM) was utilized to generate initial draft translations.
  3. Human Verification: Expert annotators with medical backgrounds meticulously corrected and verified the translations, achieving 100% inter-annotator agreement to ensure clinical accuracy.

Experimental Methodology

The researchers conducted an extensive empirical study comparing several architectural paradigms and model types:

  • Cascaded vs. End-to-End: Cascaded systems first transcribe speech to text (ASR) and then translate that text (MT). End-to-end models attempt to translate speech directly to target text.
  • Model Architectures: The study compared task-specific encoder-decoder models (e.g., mBART, M2M100, Marian) against modern, decoder-only LLMs (e.g., Llama-3.1, Qwen-2.5, Mistral).
  • Fine-tuning Strategies: They evaluated the trade-offs between bilingual fine-tuning (specializing on one language pair) and multilingual fine-tuning (training on all pairs simultaneously).

Key Insights

1. The Dominance of Cascaded Systems

A primary finding is that cascaded pipelines significantly outperform end-to-end models in the medical domain. While end-to-end models are an active area of research, they currently lack the massive amounts of parallel speech-translation data required to match the precision of a dedicated ASR model (like Whisper) paired with a robust MT model.

2. Bilingual vs. Multilingual Fine-tuning

The study revealed that bilingual fine-tuning generally yields better results than multilingual fine-tuning. In the medical context, shared parameters in multilingual models can lead to "interference" between languages with different linguistic structures, which can degrade the accuracy of specific translations.

3. Task-Specific Models vs. LLMs

Interestingly, smaller, task-specific models like M2M100 (418M parameters) often performed as well as, or better than, much larger LLMs like Llama-3.1 (8B parameters) when translating ground-truth text. This suggests that for pure translation tasks, massive generative capacity is not always a prerequisite for high performance.

4. Handling Code-Switching

Medical dialogues frequently involve "code-switching," where English terminology is interspersed with other languages. The researchers found that modern multilingual pre-trained models handle these occurrences effectively without requiring specialized architectural adjustments.

Results and Performance

The most successful configuration identified was a cascaded pipeline using Whisper-large-v2 for transcription and M2M100-418M for translation.

The study also quantified the "speech penalty"—the drop in translation quality when moving from clean text to noisy ASR transcripts. For instance, in French-to-English translation using Llama-3.1, the BLEU score fell from 50.18 on ground-truth text to 30.15 on speech-derived text. This highlights the critical importance of ASR accuracy in the overall pipeline. Furthermore, commercial baselines like Google Translate remain highly competitive, underscoring the value of the vast datasets used to train commercial systems.

Conclusion and Safety Warning

MultiMed-ST provides a vital foundation for future research in medical speech translation. However, the authors emphasize that medical translation errors carry life-altering risks. These models are research benchmarks and are not yet ready for clinical deployment. Future work must focus on closing the gap between cascaded and end-to-end systems and implementing rigorous, human-in-the-loop verification processes to ensure patient safety in real-world healthcare settings.

Summary updated

BibTeX
@misc{leduc2025multimed,
  author = {Khai Le-Duc and Tuyen Tran and Bach Phan Tat and Nam Q. Bui and Quan Anh and Hung-Phong Tran and Thanh Thuy Nguyen and Thi Ly Nguyen and Thanh Cong Phan and Thi Thu Phuong Tran and Chong‐Wah Ngo and Khanh Nguyen and Thanh Nguyen-Tang},
  title = {{MultiMed-ST: Large-scale Many-to-many Multilingual Medical Speech Translation}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.599},
}
Figure