UM-DFKI Maltese Speech Translation

Aiden Williams, Kurt Abela, Rishu Kumar, Martin Bär, Hannah Billinghurst, Kurt Micallef, Ahnaf Mozib Samin, Andrea DeMarco, Lonneke van der Plas, Claudia BorgACL 2023

TL;DR

This paper presents a cascade system for Maltese-to-English speech translation, achieving a BLEU score of 0.6. The system leverages fine-tuned XLS-R models for Automatic Speech Recognition (ASR) and mBART-50 for Machine Translation (MT), addressing the challenges of low-resource language translation.

What Problem Does This Paper Address?

Speech Translation (ST), the conversion of spoken language to written text in another language, has seen significant advancements thanks to deep learning. However, a persistent challenge lies in low-resource languages, where the scarcity of training data, linguistic expertise, and other resources hinders progress. The International Workshop on Spoken Language Translation (IWSLT) aims to foster research in this area, and this paper tackles the specific challenge of English-Maltese ST, a task that has not been previously featured in IWSLT. The authors explore existing solutions for low-resource ST and propose a pipeline system to address this gap.

Methods

The UM-DFKI team employed a cascade system, separating the Automatic Speech Recognition (ASR) and Machine Translation (MT) tasks.

  • ASR: The core of the ASR system is based on the XLS-R model, a powerful self-supervised, multilingual speech representation learning model. The authors fine-tuned five variants of the XLS-R 300M model. Notably, they experimented with fine-tuning on Maltese data alone, as well as combinations of Maltese with other languages (Arabic, French, Italian) and a model trained on all available corpora. This approach builds upon previous work on Maltese ASR by the same group.
  • MT: For Machine Translation, the system utilizes the mBART-50 model, a large multilingual sequence-to-sequence model. This model was fine-tuned on parallel Maltese-English data. The authors chose mBART-50 over mBART-25 and a baseline Transformer model due to its superior performance.

The pipeline integrates these two components: the ASR system processes the audio input, and its output text is then fed into the MT system for translation.

Datasets

  • ASR Training: The primary dataset used for fine-tuning the ASR models consisted of 50 hours of Maltese speech from a previous work (Williams, 2022). To investigate the impact of multilingual data, additional 50-hour subsets of validated data from the Common Voice corpus were used for Italian, French, and Arabic. A fifth model was trained on all these combined datasets.
  • MT Training: The machine translation system was trained on a large parallel corpus comprising over 3.6 million sentence pairs. This corpus was compiled from various publicly available sources, including Arab-Acquis, European Vaccination Portal, Publications Office of the EU, European Medicines Agency, COVID-19 ANTIBIOTIC dataset, COVID-19 EC-EUROPA, COVID-19 EU press corner V2, Digital Corpus of the European Parliament, DGT-Acquis, ELRC, Tatoeba corpus, OPUS, EUIPO - Trade mark Guidelines, Malta Government Gazette, MaCoCu, and Laws of Malta.
  • Evaluation: The official IWSLT 2023 test set was used for evaluation.

Key Insights and Ideas

  • Multilingual Fine-tuning for ASR is Detrimental: A significant finding is that fine-tuning the XLS-R model with additional languages (beyond Maltese) negatively impacted the transcription quality for Maltese. This suggests that while multilingual pre-training is beneficial, fine-tuning on mixed-language data can lead to a loss of language-specific character transcription.
  • mBART-50 Outperforms Baselines: The mBART-50 model demonstrated superior performance compared to mBART-25 and a randomly initialized Transformer model for Maltese-English translation, reinforcing its suitability for this low-resource task.
  • Challenges in Low-Resource ST: The paper highlights that even with powerful pre-trained models like XLS-R and mBART-50, low-resource ST remains a significant challenge. This is attributed to factors like the informal nature of the test data, which differs from the formal legal texts used in MT training, and potential phonetic confusions in ASR.
  • Importance of Language-Specific Tokenization: The authors noted that a standard English tokenizer struggled with Maltese, necessitating the use of a specialized tokenizer that handles Maltese-specific prefixes and articles.

Key Results

The UM-DFKI submission achieved a BLEU score of 0.6 for the Maltese-English SLT task. Among the five ASR models fine-tuned, the "MT Only" model (trained solely on Maltese data) was their primary submission. The "MT+All" model, trained on all corpora, achieved a slightly higher BLEU score of 0.7, but suffered from the issue of omitting Maltese characters. Other models trained with additional languages (MT+AR, MT+FR, MT+IT) performed worse, with BLEU scores ranging from 0.3 to 0.4.

Qualitatively, the paper points out that the test set's informal language and greetings like "merħba" (welcome) were challenging for both ASR and MT. Phonetic confusions, such as mistaking "merħba" for "erba" (four), and transcription errors like "pot kast" for "podcast" were observed.

Limitations and Future Work

  • Limitations:
    • The performance of the ASR and MT systems struggled with the official test set, indicating room for improvement.
  • The MT training data was primarily formal legal text, which did not align well with the informal nature of the test set.
  • The use of additional languages during ASR fine-tuning proved detrimental.
  • No language model was used to correct ASR output mistakes.
  • Future Work:
    • MT: Extend the mBART-50 vocabulary to include Maltese-specific tokens to improve representation. Explore techniques like back-translation and leverage monolingual corpora (e.g., Korpus Malti v4) for more data and diversity. Conduct more rigorous quality checks on parallel datasets.
  • ASR: Investigate other state-of-the-art ASR systems like Whisper and Meta's Massively Multilingual Speech (MMS) project. Focus on more targeted multilingual fine-tuning. Explore transliteration of foreign alphabetic characters into Maltese. Implement a language model to correct ASR errors.

Summary updated

BibTeX
@inproceedings{williams2023um,
  author = {Aiden Williams and Kurt Abela and Rishu Kumar and Martin Bär and Hannah Billinghurst and Kurt Micallef and Ahnaf Mozib Samin and Andrea DeMarco and Lonneke van der Plas and Claudia Borg},
  title = {{UM-DFKI Maltese Speech Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure