NICT’s Cascaded and End-To-End Speech Translation Systems using Whisper and IndicTrans2 for the Indic Task

Raj Dabre, Haiyue SongACL 2024

TL;DR

This paper tackles the challenge of low-resource speech-to-text translation for Indic languages by leveraging powerful pre-trained models. They propose both a cascaded system (ASR then MT) and an end-to-end system, finding that the cascaded approach significantly outperforms the end-to-end model, highlighting the ongoing importance of robust ASR in this domain.

What Problem Does This Paper Address?

Speech-to-text (ST) translation is vital for breaking down language barriers, especially in global communication. While ST systems perform well for high-resource language pairs, they struggle with low-resource languages like Hindi, Bengali, and Tamil. This is primarily due to a lack of sufficient parallel data for training robust models. This paper addresses this data scarcity challenge for English-to-Indic language ST by exploring two system architectures: a cascaded approach and an end-to-end approach, both built upon state-of-the-art pre-trained models.

Methods

The authors explore two main system architectures:

  • Cascaded System: This system follows a traditional two-stage pipeline. First, a Whisper model, fine-tuned for Automatic Speech Recognition (ASR), transcribes English speech into English text. Second, an IndicTrans2 model, fine-tuned for Machine Translation (MT), translates the English text into the target Indic language. The key here is fine-tuning both Whisper and IndicTrans2 on the available Indic language data.
  • End-to-End (E2E) System: This approach aims to directly translate speech to text in a single model. To overcome the data scarcity issue for E2E training, they employ a "pseudo translation data" technique, essentially knowledge distillation. They first use the fine-tuned IndicTrans2 model to generate translations of English text into Indic languages. These generated translations, which are more consistent than human-annotated transcriptions, are then used to fine-tune the Whisper model for direct speech-to-Indic translation.

Datasets

The primary dataset used in this work is the official corpus provided for the IWSLT 2024 Indic track. This corpus includes English speech with corresponding transcriptions and translations in Hindi, Bengali, and Tamil. The paper notes that they did not leverage any additional datasets to avoid potential test set overlap issues.

Key Insights and Ideas

  • Cascaded systems still reign supreme for low-resource ST: Despite advancements in end-to-end models, the cascaded approach, leveraging a strong ASR component, significantly outperforms the E2E system for English-to-Indic language translation. This underscores the critical role of accurate speech recognition in low-resource scenarios.
  • Knowledge distillation helps E2E, but not enough to surpass cascaded: While using pseudo-translations from a strong MT model (IndicTrans2) to train the E2E Whisper model offers a noticeable improvement (around 0.7 BLEU), it still falls far short of the cascaded system's performance.
  • Fine-tuning pre-trained models is crucial: The paper demonstrates that fine-tuning both Whisper for ASR and IndicTrans2 for MT on the target Indic language data leads to substantial gains compared to using them off-the-shelf.
  • Domain mismatch in spoken language: The authors observe a format mismatch between spoken language (less punctuation) and written language, which can affect ASR performance. They found that fine-tuning Whisper helped mitigate this issue.

Key Results

  • The cascaded system achieved a BLEU score of 51.0 on the test set.
  • The end-to-end system scored 19.1 BLEU on the test set.
  • The cascaded system outperformed the end-to-end system by approximately 32 BLEU points.
  • Knowledge distillation in the E2E system improved translation quality by about 0.7 BLEU points compared to using gold standard translations.

Limitations and Future Work

  • Limitations: The primary limitation identified is the inherent data scarcity for Indic languages, which significantly hinders the performance of end-to-end systems. The paper also notes a format mismatch issue between spoken and written language, though they found fine-tuning Whisper helped.
  • Future Work: The authors plan to focus on combining Whisper and IndicTrans2 jointly to train an even stronger speech translation system, suggesting a move towards more integrated architectures in the future.

Summary updated

BibTeX
@inproceedings{dabre2024nicts,
  author = {Raj Dabre and Haiyue Song},
  title = {{NICT’s Cascaded and End-To-End Speech Translation Systems using Whisper and IndicTrans2 for the Indic Task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure