JHU IWSLT 2025 Low-resource System Description

Nathaniel Romney Robinson, Niyati Bafna, Xuan He, Tom Lupicki, Lavanya Shankar, Cihan Xiao, Qi Sun, Kenton Murray, David Yarowsky2025 · DOI

Abstract

Nathaniel Romney Robinson, Niyati Bafna, Xiluo He, Tom Lupicki, Lavanya Shankar, Cihan Xiao, Qi Sun, Kenton Murray, David Yarowsky. Proceedings of the 22nd International Conference on Spoken Language Translation (IWSLT 2025).

2025.

Introduction

Speech Translation (ST) for low-resource languages, such as Fongbe, Bemba, or Quechua, presents significant challenges due to the scarcity of training data. Models in these settings often exhibit high variance, where one system might produce a high-quality translation while another generates hallucinations. The Johns Hopkins University (JHU) team addressed these challenges for the IWSLT 2025 Low-Resource Speech Translation task by exploring ensembling strategies to bridge the performance gap across typologically diverse languages.

Methodology

The JHU system leverages a "zoo" of diverse model architectures combined through a sophisticated selection process.

Diverse Architectures and Training

The team utilized three primary models: SeamlessM4T v2 (an end-to-end multilingual model), Whisper (used for both ASR and end-to-end translation), and NLLB (No Language Left Behind) for machine translation. The experimental setup compared cascaded systems (ASR followed by MT) against direct end-to-end (E2E) models. To improve performance, they employed intra-distillation for regularization and mixed data training, grouping related languages to encourage cross-lingual generalization.

Minimum Bayes Risk (MBR) Decoding

The core of the ensembling strategy is MBR decoding. Instead of selecting the single most likely output from one model, the system samples multiple hypotheses from various model checkpoints. MBR then identifies the "central" candidate—the one with the highest average similarity (measured by BLEU or chrF) to all other candidates. This approach minimizes the risk of selecting an outlier translation.

Figure 1: Overview of the JHU ensemble translation system using MBR decoding across multiple model checkpoints.

Datasets

The team evaluated their approach on 10 language pairs provided by IWSLT 2025, including Levantine Arabic, Tunisian Arabic, Bemba, Fongbe, Irish, Bhojpuri, Estonian, Maltese, Marathi, and Quechua. Data sources included organizer-provided sets and external corpora like Common Voice and the 1,280-hour Shrutilipi Marathi ASR dataset.

Results and Discussion

The results across the 10 language pairs highlighted both the strengths and limitations of ensembling in low-resource contexts.

Figure 4: BLEU scores for various systems across the 10 IWSLT 2025 low-resource language pairs.

The "Safety Net" Effect

MBR ensembling acted as a reliable "safety net." While it did not always produce the highest possible score compared to the best-performing standalone model for a specific language, it consistently reached "near-best" performance. This is particularly valuable when the optimal model architecture for a given language is unknown a priori.

Key Findings

  • Metric Robustness: Using chrF as the similarity metric for MBR frequently outperformed BLEU-based MBR, indicating that character-level similarity is more robust for low-resource evaluation.
  • Training Duration: The team found that dev scores for SeamlessM4T v2 continued to improve up to 10 epochs, significantly longer than the 4 epochs typically suggested in literature.
  • Performance Variance: Results ranged from a high of 56.80 BLEU for Maltese-to-English to a low of 11.0 BLEU for Quechua-to-Spanish, underscoring the extreme difficulty of the latter.

Conclusion

While ensembling provides a robust framework for low-resource ST, the JHU team observed diminishing returns. In some instances, the ensemble was "pulled" toward a mediocre consensus by lower-quality candidates. Future research should focus on more innovative applications of ensembling and multilingual training rather than simply increasing the number of models in the pool.

Summary updated

BibTeX
@misc{robinson2025jhu,
  author = {Nathaniel Romney Robinson and Niyati Bafna and Xuan He and Tom Lupicki and Lavanya Shankar and Cihan Xiao and Qi Sun and Kenton Murray and David Yarowsky},
  title = {{JHU IWSLT 2025 Low-resource System Description}},
  year = {2025},
  doi = {10.18653/v1/2025.iwslt-1.32},
}
Figure