Abstract
Nathaniel Romney Robinson, Kaiser Sun, Cihan Xiao, Niyati Bafna, Weiting Tan, Haoran Xu, Henry Li Xinyuan, Ankur Kejriwal, Sanjeev Khudanpur, Kenton Murray, Paul McNamee. Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024).
2024.
Introduction: The Low-Resource Challenge
While speech translation (ST) has achieved remarkable performance in high-resource language pairs, such as English-to-Spanish, low-resource languages and dialects remain significantly underserved. These languages often exist in a "data desert," lacking the massive parallel audio-text datasets typically required to train modern Transformer-based models from scratch.
In this work, the Johns Hopkins University (JHU) team investigated strategies to adapt large-scale "foundation models"—pre-trained on thousands of hours of diverse audio—to eight specific low-resource language pairs. These included dialects like Levantine Arabic and indigenous languages such as Quechua and Bemba. A central research question was whether an end-to-end (E2E) approach (mapping audio directly to translated text) or a cascaded approach (transcription followed by machine translation) is more effective when training data is extremely limited.
System Architectures and Methodologies
The researchers evaluated three primary architectural frameworks to address the translation task:
- End-to-End SeamlessM4T v2: This involves fine-tuning Meta’s 2.3-billion parameter multimodal model.
- End-to-End Whisper: OpenAI’s Whisper (Large-v2) was fine-tuned using multi-task learning (MTL) to simultaneously handle Automatic Speech Recognition (ASR) and Speech Translation (ST).
- Cascaded Systems: A pipeline approach where a fine-tuned Whisper model performs ASR, followed by a fine-tuned No Language Left Behind (NLLB) model for Machine Translation (MT).
Beyond standard fine-tuning, the team introduced several specialized techniques:
- Intra-distillation (ID): Applied during NLLB fine-tuning, this technique encourages different parameter subsets to produce consistent outputs, enhancing generalization on small datasets.
- Pseudo-translation: For linguistically similar pairs like Marathi/Bhojpuri to Hindi, Whisper was prompted to "transcribe" source audio directly into the target language text, treating translation as a variant of transcription.
- Curriculum Learning: For Maltese, the model was first fine-tuned on Tunisian Arabic (a related Semitic language) before training on the target Maltese data.
Data Resources
The study encompassed eight language pairs, including translations to English (Levantine Arabic, Bemba, Irish, Maltese), Hindi (Bhojpuri, Marathi), Spanish (Quechua), and French (Tamasheq). The team utilized IWSLT-provided datasets supplemented by external sources such as Common Voice and ParaCrawl. Data availability was notably sparse, ranging from a mere 1.7 hours for Quechua to 180 hours for Bemba.
Experimental Results
The performance of the systems varied significantly based on the language pair and the amount of pre-training data available for those specific languages in the foundation models.
Key quantitative findings include:
- Marathi to Hindi: SeamlessM4T v2 achieved a robust BLEU score of 37.7, benefiting from the model's prior exposure to these languages.
- Maltese to English: SeamlessM4T v2 dominated this track with a BLEU of 54.2.
- Bemba to English: The cascaded system (Whisper+NLLB+ID) reached 32.6 BLEU, significantly outperforming the E2E Seamless model, which only achieved 15.4.
- Intra-distillation: The inclusion of ID in the NLLB module provided consistent performance gains, particularly for the Quechua-to-Spanish task.
Discussion and Analysis
Several critical insights emerged from the experiments:
- Pre-training Bias: SeamlessM4T v2 excelled on languages included in its original pre-training (e.g., Marathi, Maltese) but struggled with "unseen" languages like Bemba or Quechua.
- Cascade Robustness: For languages absent from foundation model pre-training, the cascaded approach (Whisper + NLLB) was superior. Decomposing the task into two simpler steps (Audio to Text and Text to Translation) proved more resilient than direct E2E mapping.
- Multi-task Benefits: Training Whisper on both ASR and ST tasks simultaneously improved overall performance. The ASR task serves as a regularizer, helping the model maintain high-quality internal audio representations.
- Linguistic Proximity: The "pseudo-translation" strategy for Marathi/Bhojpuri to Hindi demonstrated that linguistic similarity can be effectively exploited to mitigate data scarcity.
Conclusion and Future Directions
The JHU team's participation in IWSLT 2024 highlights that there is no universal architecture for low-resource speech translation. While E2E models are powerful when pre-trained data is available, cascaded systems remain a vital tool for truly underserved languages.
Future work will focus on addressing error propagation in cascaded systems, where ASR errors lead to "hallucinated" translations. The researchers also suggest exploring Self-Supervised Learning Representations (SSLR) like wav2vec2 for even more data-starved scenarios and utilizing back-translation to synthetically augment training sets.
Summary updated