BUT Systems for IWSLT 2023 Marathi - Hindi Low Resource Speech Translation Task

Santosh Kesiraju, Karel Beneš, Maksim Tikhonov, Jan ČernockýACL 2023

TL;DR

This paper presents two systems for Marathi-Hindi low-resource speech translation, one end-to-end and one cascaded. Both leverage a strong bilingual ASR model and explore transfer learning, data augmentation, and joint decoding to achieve competitive results on a challenging task with limited parallel data.

What Problem Does This Paper Address?

Speech translation (ST) systems typically require large amounts of parallel data (source speech, source transcription, target text). However, for many language pairs, especially those considered "low-resource," such data is scarce or unavailable. This paper tackles the Marathi-Hindi low-resource ST task, where direct speech-to-translation pairs are limited, and source transcriptions are absent. The challenge lies in effectively utilizing available resources, such as large amounts of transcribed speech for Automatic Speech Recognition (ASR) and text-only translation data, to build a functional ST system.

Methods

The authors developed two main system architectures:

  • End-to-End (E2E) System: This is their primary submission. It's built upon a pre-trained bilingual ASR model. The ASR encoder and decoder are repurposed for ST. Crucially, they employ a joint CTC (Connectionist Temporal Classification) and attention objective during training, which helps the model learn to directly map speech to target text. This is further enhanced by joint decoding, where CTC and attention probabilities are combined. They also explore data augmentation via speed perturbation and external language model (LM) re-scoring of n-best hypotheses.
  • Cascaded System: This system acts as a contrastive approach. It also uses the pre-trained bilingual ASR model as a backbone. The ASR output (hypotheses) is then fed into a separate Machine Translation (MT) model. This MT model is initialized from a cross-lingual language model (XLM). Finally, an external LM is used to re-score the MT output.

Both systems rely on "backbone models" (ASR, MT, LM) trained on publicly available data.

Datasets

The paper details the datasets used for training the backbone models:

  • Paired Speech Data (for ASR):
    • GramVaani (GV): Hindi telephone quality speech.
  • Indian Language Corpora (ILC): Marathi speech with transcriptions.
  • Mozilla Common Voice v12 (MCV): Hindi and Marathi validated speech.
  • MUCS: Hindi and Marathi multilingual and code-switched corpus.
  • Multi-speaker speech corpora (MSSC): Marathi speech.
  • Shrutilipi (SL): Large collection of radio broadcast news in various Indian languages (Hindi and Marathi filtered).
  • Monolingual and Parallel Text Data (for MT and LM):
    • Monolingual data pooled from transcribed speech, Samanantar, Indic2Indic, and IIIT-H CVIT corpora.
  • Parallel text from Indic2Indic split of Samanantar.
  • Speech Translation Data (for ST fine-tuning):
    • Official IWSLT 2023 Marathi-Hindi data (approx. 16 hours of Marathi speech with Hindi translations, no source transcriptions). Speed perturbation was used for augmentation.

Key Insights and Ideas

  • Leveraging ASR for ST: Pre-training a strong bilingual ASR model is highly effective for initializing ST systems, especially in low-resource settings. The ASR encoder's representations are well-suited for the ST decoder.
  • Joint CTC and Attention: Combining CTC loss with the attention objective in the E2E ST model significantly improves performance. CTC provides a non-autoregressive path, while attention handles the sequence-to-sequence mapping.
  • Data Augmentation Matters: Speed perturbation on the limited ST training data provided noticeable improvements.
  • External LM Re-scoring: Using an external LM to re-score the n-best hypotheses from the ST system further boosted performance, particularly for the E2E approach.
  • Cascaded vs. E2E Trade-offs: While both approaches yielded competitive results, the E2E system generally outperformed the cascaded one, suggesting the benefits of direct mapping.

Key Results

  • Primary Submission (E2E): Achieved 39.6 BLEU and 63.3 CHRF2 on the official test set. This system was based on a bilingual ASR model (B2), fine-tuned with data augmentation and LM re-scoring.
  • Contrastive System (Cascaded): Achieved 28.6 BLEU and 54.4 CHRF2 on the official test set. This system used the same backbone ASR (B2) and an MT model, followed by LM re-scoring.
  • ASR Performance: Bilingual ASR models (B1, B2) consistently outperformed monolingual ASR models.
  • CTC Weight Optimization: An optimal CTC weight of 0.3 was found for both joint training and decoding in the E2E system.

Limitations and Future Work

  • Data Scarcity: The primary limitation remains the scarcity of parallel speech translation data.
  • LM Performance in Cascade: The cascaded system's LM re-scoring did not yield improvements, potentially due to the Marathi LM being trained on a smaller dataset (400K sentences).
  • Text Normalization: Manual inspection revealed issues with ambiguous numerals in translations. Further text normalization of both references and hypotheses could improve evaluation.
  • Future Work: The authors suggest investigating if large pre-trained MT systems could close the gap between cascaded and E2E systems. They also plan to re-run experiments with improved LM data and explore other hyperparameter tuning strategies.

Summary updated

BibTeX
@inproceedings{kesiraju2023but,
  author = {Santosh Kesiraju and Karel Beneš and Maksim Tikhonov and Jan Černocký},
  title = {{BUT Systems for IWSLT 2023 Marathi - Hindi Low Resource Speech Translation Task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure