JHU IWSLT 2022 Dialect Speech Translation System Description

Jinyi Yang, Amir Hussein, Matthew Wiesner, Sanjeev KhudanpurACL 2022

TL;DR

This paper describes the Johns Hopkins University system for the IWSLT 2022 Dialect Speech Translation task, focusing on Tunisian Arabic. They found that a cascaded ASR-MT system outperformed end-to-end models, and that aggressive text normalization and back-translation techniques were crucial for improving performance on this challenging dialect.

What Problem Does This Paper Address?

The paper tackles the significant challenge of speech translation (ST) for non-standard dialects, specifically Tunisian Arabic. Unlike dominant dialects, these often lack formal orthography, grammar, and extensive data resources, making traditional ST approaches difficult. The core problem is how to leverage existing resources from standard dialects (like Modern Standard Arabic - MSA) to build effective ST systems for these under-resourced dialects, especially when both Automatic Speech Recognition (ASR) and Machine Translation (MT) components are inherently less reliable for unwritten or inconsistently written dialects.

Methods

The JHU system employs a cascaded ASR-MT architecture, where input audio is first transcribed by an ASR model, and then the resulting text is translated by an MT model. This contrasts with end-to-end models that directly translate speech to text.

Key methodological contributions include:

  • ASR:
    • For the "Basic" condition, they trained on Tunisian Arabic audio and transcripts.
  • For "Dialect Adaptation," they leveraged large-scale MSA data (MGB-2) to pre-train a Conformer model, then fine-tuned it on Tunisian data. They explored different domain matching strategies for pre-training and fine-tuning.
  • For "Unconstrained," they fine-tuned the multilingual Wav2Vec2 XLSR-53 model, which they hypothesized was a better starting point due to its training on diverse Arabic and French data.
  • MT:
    • Transformer architectures were used for MT models.
  • For "Basic" and "Dialect Adaptation," they trained models on the respective available data.
  • A significant contribution was the use of back-translation to mitigate dialect mismatch. This involved training an English-to-MSA model, translating English data to MSA, and then training an MSA-to-Tunisian model. This synthetic Tunisian data was then used to train a Tunisian-to-English MT model.
  • Text Normalization: Aggressive normalization of Tunisian Arabic transcripts was applied, including removing punctuation, converting digits, removing diacritics, normalizing Alif/Ya/Ta-Marbuta, and handling single character words.

Datasets

  • Tunisian Arabic Speech Translation Data: Provided by the organizers, this dataset includes audio, Tunisian Arabic transcripts, and English translations for training, development, and testing.
  • Modern Standard Arabic (MSA) Data: Used for pre-training ASR and MT models. Specifically, the MGB-2 dataset (broadcast news) was utilized.
  • English-MSA Bitext: Large amounts of MSA-English bitext from OPUS were used for MT model adaptation.

Key Insights and Ideas

  • Cascaded Systems Outperform End-to-End: Surprisingly, the cascaded ASR-MT system achieved better performance than end-to-end models, even with a substantial amount of training data for the latter. This suggests that modularity and the ability to leverage separate ASR and MT resources are still highly beneficial for dialectal speech translation.
  • Text Normalization is Crucial: Aggressive text normalization of the Tunisian Arabic transcripts yielded significant improvements in MT performance. This highlights the importance of standardizing noisy or varied dialectal text before translation.
  • Back-Translation Bridges Dialect Gaps: The back-translation approach, which generates synthetic Tunisian data from MSA, proved effective in reducing dialect mismatch and improving MT performance, even outperforming models trained directly on MSA data.
  • MSA Pre-training Benefits MT: Pre-training MT models on large MSA-English bitext and then fine-tuning on Tunisian-English data led to substantial improvements.

Key Results

  • The best ST performance was achieved using the cascaded system.
  • Aggressive text normalization improved MT performance significantly.
  • The back-translation approach (Ta2En-bt-tune) achieved the best MT performance, showing an average of ~8 BLEU points improvement over models not using this technique.
  • For ASR, the MGB2-tune-best model (conformer pre-trained on down-sampled 8KHz MGB-2 and fine-tuned on Tunisian data) achieved the best performance on the Tunisian Arabic test set.
  • The Wav2Vec2 XLSR-53 model, while showing some promise for unconstrained ASR, yielded only marginal gains over baseline models.

Limitations and Future Work

  • End-to-End System Diagnosis: The paper notes that the poorer performance of end-to-end systems compared to cascaded ones warrants further investigation.
  • Back-Translation Optimization: Future work should focus on determining the optimal method for minimizing dialect mismatch when augmenting training data with back-translation.
  • "Real" Evaluation of MSA-TA Model: The authors plan to conduct a "real" evaluation of their MSA-to-Tunisian model using ground-truth MSA-Tunisian data, rather than synthetic MSA data, in future work.
  • ASR Model for Submission: The best performing ASR model (MGB2-tune-best) was only completed after the evaluation period, so it was not used for the final submission. This suggests potential for further improvement if this model had been available.

Summary updated

BibTeX
@inproceedings{yang2022jhu,
  author = {Jinyi Yang and Amir Hussein and Matthew Wiesner and Sanjeev Khudanpur},
  title = {{JHU IWSLT 2022 Dialect Speech Translation System Description}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure