QUESPA Submission for the IWSLT 2024 Dialectal and Low-resource Speech Translation Task

John E. Ortega, Rodolfo Joel Zevallos, Ibrahim Said Ahmad, William ChenACL 2024

TL;DR

The QUESPA team presents their approach to the IWSLT 2024 Quechua-Spanish speech translation task, focusing on low-resource and dialectal challenges. They achieved their best results by fine-tuning the SpeechT5 model with data augmentation, demonstrating the power of pre-trained models and advanced techniques for low-resource languages.

What Problem Does This Paper Address?

Speech translation (ST) is notoriously difficult for low-resource languages due to the scarcity of parallel data. Traditionally, this has been tackled by cascading Automatic Speech Recognition (ASR) and Machine Translation (MT) systems, leveraging advances in pre-trained language models (PLMs). However, creating effective models for languages with limited data, like Quechua, remains a significant challenge. This paper addresses the specific problem of building robust Quechua-to-Spanish ST systems within the constrained and unconstrained settings of the IWSLT 2024 evaluation, aiming to improve upon previous benchmarks.

Methods

The QUESPA team employed a combination of established and novel techniques for their ST systems. For the constrained setting, they utilized a direct ST approach based on the Fairseq S2T framework, employing log mel-scale filter banks for audio features and a transformer architecture. They experimented with different model sizes ("extra-small" and "medium" transformers) and explored data augmentation techniques.

In the unconstrained setting, their primary system leveraged the pre-trained SpeechT5 model, fine-tuned for ST. This involved multi-modal pre-training on LibriSpeech and subsequent fine-tuning on Quechua-Spanish data. They also incorporated data augmentation techniques, including nlpaug and DA-TTS, to expand the training data. A contrastive system explored using the Whisper ASR model in a cascaded ASR+MT setup with the NLLB MT system.

Datasets

The primary dataset used is the Quechua-Spanish (QUE-SPA) low-resource ST track dataset from IWSLT 2024. This dataset includes:

  • 1 hour and 40 minutes of constrained speech with corresponding translations.
  • Approximately 48 hours of ASR data (with transcriptions) from the Siminichik corpus.
  • A Quechua-Spanish MT dataset from previous neural MT work, comprising 100 daily magazine articles and 51,140 religious sentences.

For the unconstrained setting, they also utilized:

  • The 48 hours of audio provided by the organizers.
  • A portion of the AmericasNLP 2022 speech translation competition corpus, including 19 minutes of Guarani and 29 minutes of Bribri, translated into Spanish.

Key Insights and Ideas

  • Pre-trained models are crucial for low-resource ST: The success of their unconstrained systems, particularly the SpeechT5-based one, highlights the significant advantage gained from leveraging large, pre-trained models.
  • Data augmentation is a powerful tool: Even with limited data, applying various data augmentation techniques (e.g., noise, roll, time, pitch, nlpaug, DA-TTS) significantly boosted performance, especially in the unconstrained task.
  • Model size matters, but not always: While they explored different transformer sizes, the "extra-small" transformer performed best in the constrained setting, suggesting that larger models aren't always superior for very limited data.
  • Constrained ST remains challenging: Despite improvements, achieving high BLEU scores in the constrained setting proved difficult, indicating ongoing research needs in this area.
  • Dialectal unification is beneficial: Combining Quechua I and Quechua II dialects into a single dataset for training was a key strategy.

Key Results

  • Constrained Setting: The best performing constrained system achieved a BLEU score of 1.96. This represents a notable improvement over previous years, with data augmentation showing promise.
  • Unconstrained Setting:
    • The primary system achieved a BLEU score of 16.0.
  • The contrastive 1 system, utilizing SpeechT5 with extensive data augmentation, achieved the highest BLEU score of 19.7.
  • The contrastive 2 system (Whisper ASR + NLLB MT) scored 11.1 BLEU.

Limitations and Future Work

  • Constrained Setting Performance: The authors acknowledge that constrained systems still struggle to achieve deployable performance, with BLEU scores remaining low even with improvements.
  • Data Mismatch: They observed an uneven distribution in the training and validation/test sets, with shorter utterances in training and longer ones in validation/test, which posed a challenge.
  • Future Work: The team plans to further explore data augmentation techniques, including features like SpecAugment and roll addition, for both constrained and unconstrained tasks in future IWSLT challenges. They also aim to experiment with different feature sets.

Summary updated

BibTeX
@inproceedings{ortega2024quespa,
  author = {John E. Ortega and Rodolfo Joel Zevallos and Ibrahim Said Ahmad and William Chen},
  title = {{QUESPA Submission for the IWSLT 2024 Dialectal and Low-resource Speech Translation Task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure