TL;DR
This paper details UoM-DFKI's submission to the IWSLT 2024 low-resource speech translation task for Maltese to English. They explored two end-to-end approaches, a primary system using the large Whisper model and a contrastive system based on wav2vec 2.0 with mBART decoders, achieving a strong primary result of 35.1 BLEU.
What Problem Does This Paper Address?
The paper tackles the challenge of speech translation for low-resource languages, specifically Maltese to English. Maltese is an interesting case due to its hybrid vocabulary, drawing heavily from Arabic, Italian, and English, yet it uses the Latin script. This presents an opportunity to test the cross-lingual capabilities of modern end-to-end speech translation models, which often struggle with languages that have limited parallel data. The IWSLT shared task aims to push the boundaries of translation quality for such languages.
Methods
The UoM-DFKI team employed two primary end-to-end approaches:
- Whisper-based System (Primary): They fine-tuned the largest available Whisper model (Whisper-large-v3) on the provided Maltese speech translation dataset. Whisper is pre-trained on a massive amount of speech data, including a significant portion of English and Maltese, making it a strong candidate for this task.
- wav2vec 2.0-based System (Contrastive): This system utilized a SpeechEncoderDecoder framework. The encoder was based on wav2vec 2.0 (specifically, xls-r with 2B parameters), and they experimented with three different decoders: a base BERT model, and mBART fine-tuned for machine translation. This approach aimed to provide a contrast to the more monolithic Whisper model.
Both systems involved pre-processing steps, including converting numbers to words and removing punctuation and capitalization from the input text.
Datasets
The primary dataset used for fine-tuning was derived from the IWSLT 2024 shared task. This dataset was constructed from subsets of the Common Voice and MASRI corpora, specifically:
- Common Voice: 5 hours and 11 minutes of verified data.
- MASRI-Headset: 6 hours and 39 minutes of data.
The transcriptions for these audio samples were translated. The paper also references the ML50 dataset used in prior work for fine-tuning mBART50 models, which leverages English as a pivot language to gather parallel data for 49 other languages.
Key Insights and Ideas
- Whisper's Dominance: The large, pre-trained Whisper model significantly outperformed the more modular wav2vec 2.0-based system, highlighting the power of massive, self-supervised pre-training for end-to-end speech translation.
- The Challenge of Contrastive Systems: The contrastive system, while designed to offer a different perspective, struggled, particularly with the MASRI subset of the data. This suggests that the specific architecture and training of the encoder-decoder approach, or perhaps the data characteristics, were less effective for Maltese.
- Data Quality Matters: The paper notes that the MASRI corpus provided cleaner audio than Common Voice, and this difference seemed to impact the contrastive system's performance.
- Repetitive Errors: A manual analysis of the contrastive system revealed a recurring issue of repeated phrases or sentences in bad translations, a phenomenon that proved difficult to debug and resolve within the scope of the study.
- Potential for Data Augmentation: The authors hypothesize that data augmentation and combining parallel data from related languages like Arabic and Italian could improve the performance of their encoder-decoder models.
Key Results
- Primary System (Whisper): Achieved a BLEU score of 35.1 on the combined test set.
- Contrastive System (wav2vec 2.0 + mBART): Achieved a BLEU score of 18.5 on the combined test set.
- The Whisper-based system consistently outperformed the contrastive system across different subsets of the data.
- The performance on the test set closely mirrored the results obtained on the development set for both systems.
Limitations and Future Work
- Contrastive System Performance: The authors acknowledge the significantly lower performance of their contrastive system, especially on the MASRI subset, and attribute it to potential issues with data quality or the model's architecture.
- Repetitive Translation Errors: The issue of repeated phrases in the contrastive system's output was not fully resolved, and the authors suggest exploring post-fix approaches for future work.
- Data Augmentation and Multi-lingual Data: The paper suggests that data augmentation and incorporating parallel data from Arabic and Italian could enhance the performance of encoder-decoder models.
- Investigating mBART for Direct Translation: They plan to analyze the performance of a fine-tuned mBART50 model used directly for translation from Arabic and Italian to English, comparing it to its use as a decoder.
- Pipeline-based Approaches: The authors are interested in exploring pipeline-based approaches using Whisper-based models.
- Language-Specific Adapters: They propose exploring the use of language-specific adapters for Maltese-English SLT, drawing inspiration from successful applications in other low-resource contexts.
Summary updated