ON-TRAC Consortium Systems for the IWSLT 2022 Dialect and Low-resource Speech Translation Tasks

Marcely Zanon Boito, John Ortega, Hugo Riguidel, Antoine Laurent, Loïc Barrault, Fethi Bougares, Firas Chaabani, Ha Nguyen, Florentin Barbier, Souhir Gahbiche, Yannick Estève

TL;DR

The ON-TRAC Consortium explores the frontier of low-resource speech translation (ST) for the IWSLT 2022 challenge, focusing on Tunisian Arabic and Tamasheq. Their findings reveal that cascaded pipeline systems remain surprisingly competitive against end-to-end models when leveraging related high-resource data, and that domain-specific self-supervised models often outperform massive, general-purpose ones in data-scarce settings.

What Problem Does This Paper Address?

Most modern speech processing pipelines are built for high-resource languages like English or Mandarin, where annotated data is abundant. However, these models often fail in "real-world" scenarios involving low-resource languages or dialects.

The authors tackle two specific hurdles:

  1. Dialectal Variation: Tunisian Arabic lacks a standard written form and varies significantly from Modern Standard Arabic (MSA), making traditional ASR and translation difficult.
  2. Extreme Data Scarcity: Tamasheq (a Tuareg language) has very little translated data and, in this challenge, lacked any transcriptions at all, preventing standard supervised training for Automatic Speech Recognition (ASR).

Methods

The team investigated two primary architectural philosophies: Cascaded Pipelines (ASR followed by Machine Translation) and End-to-End (E2E) Models (direct audio-to-translated-text).

Tunisian Arabic-English

  • Transfer Learning: They pre-trained an E2E ASR model on 820 hours of MSA data (MGB2 dataset) before fine-tuning it on Tunisian Arabic.
  • Cascaded vs. E2E: They compared a Conformer-based E2E ST model against pipelines using various MT backends (bi-LSTM and fully convolutional).

Tamasheq-French

  • SSL Leveraging: They used wav2vec 2.0 models. Critically, they found that reducing the number of encoder layers (the "W2V-N+ST" approach) helped the model converge better on the small 17-hour Tamasheq dataset.
  • Pseudo-Phonetic Transcriptions: Since Tamasheq had no transcripts, they used a French ASR system to generate "approximate" phonetic transcriptions. They then used these "fake" transcripts to train a multi-task model that jointly optimized for ASR, MT, and ST losses.

Datasets

  • Tunisian Arabic: LDC2022E01 (383h of speech, 160h translated to English).
  • MSA (External): MGB2 dataset (1,200h of broadcast news) used for pre-training Tunisian ASR.
  • Tamasheq-French: A recent corpus containing 17h of translated speech.
  • Niger-Mali Collection: 641h of unlabeled audio in Tamasheq and geographically related languages (Fulfulde, Hausa, Zarma) used for self-supervised pre-training.

Key Insights and Ideas

  • Bigger isn't always better: In the Tamasheq experiments, massive off-the-shelf models like XLSR-53 (trained on 56k hours) performed poorly. Smaller "base" models pre-trained specifically on Tamasheq or related regional languages were far more effective.
  • The "Cascaded" Comeback: For Tunisian Arabic, the cascaded systems actually outperformed the E2E primary submission. The authors attribute this to the ability of the cascaded pipeline to better ingest the external 820 hours of MSA data during the ASR stage.
  • Phonetic Proxies Work: Even though the French-based phonetic transcriptions for Tamasheq were "approximate," they provided enough signal to allow multi-task learning to significantly boost translation scores compared to a baseline pipeline.
  • Layer Pruning: For E2E ST fine-tuning, dropping the final layers of the wav2vec 2.0 encoder (keeping only the first 6 of 12) yielded the best results, suggesting that lower-level acoustic features are more transferable than high-level task-specific ones in low-resource settings.

Key Results

For Tunisian Arabic, the post-evaluation cascaded model achieved the highest BLEU score of 14.4, outperforming the primary E2E model's 12.4.

Table 2: Tunisian Arabic to English translation results

In the Tamasheq-French task, the E2E model utilizing the pruned wav2vec 2.0 encoder (W2V-6+ST) achieved a BLEU of 5.70, a massive improvement over the simple pipeline baseline of 1.80.

Table 3: Tamasheq-French ST system results

Limitations and Future Work

The authors noted that off-the-shelf zero-shot models (like XLS-R) suffered from severe "repetition" issues during decoding, sometimes repeating phrases hundreds of times.

A major limitation was hardware: training large E2E models required significant GPU memory, forcing a reduction in decoder size that may have hampered performance. Future work will focus on combining the primary SSL-based speech encoder with the multi-task "pseudo-phonetic" framework to see if the benefits of both approaches are additive.

Summary updated

Abstract

This paper describes the ON-TRAC Consortium translation systems developed for two challenge tracks featured in the Evaluation Campaign of IWSLT 2022: low-resource and dialect speech translation. For the Tunisian Arabic-English dataset (low-resource and dialect tracks), we build an end-to-end model as our joint primary submission, and compare it against cascaded models that leverage a large fine-tuned wav2vec 2.0 model for ASR. Our results show that in our settings pipeline approaches are still very competitive, and that with the use of transfer learning, they can outperform end-to-end models for speech translation (ST). For the Tamasheq-French dataset (low-resource track) our primary submission leverages intermediate representations from a wav2vec 2.0 model trained on 234 hours of Tamasheq audio, while our contrastive model uses a French phonetic transcription of the Tamasheq audio as input in a Conformer speech translation architecture jointly trained on automatic speech recognition, ST and machine translation losses. Our results highlight that self-supervised models trained on smaller sets of target data are more effective to low-resource end-to-end ST fine-tuning, compared to large off-the-shelf models. Results also illustrate that even approximate phonetic transcriptions can improve ST scores.
BibTeX
@inproceedings{boito2022trac,
  author = {Marcely Zanon Boito and John Ortega and Hugo Riguidel and Antoine Laurent and Loïc Barrault and Fethi Bougares and Firas Chaabani and Ha Nguyen and Florentin Barbier and Souhir Gahbiche and Yannick Estève},
  title = {{ON-TRAC Consortium Systems for the IWSLT 2022 Dialect and Low-resource Speech Translation Tasks}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure