JHU IWSLT 2023 Multilingual Speech Translation System Description

Henry Li Xinyuan, Neha Verma, Bismarck Bamfo Odoom, Ujvala Pradeep, Matthew Wiesner, Sanjeev KhudanpurACL 2023

TL;DR

The Johns Hopkins University team developed multilingual speech translation systems for the IWSLT 2023 task, translating academic talks from English into 10 languages. They explored both constrained and unconstrained approaches, leveraging pre-trained models and creating a domain-specific corpus to tackle the challenges of technical jargon and diverse speaker accents.

What Problem Does This Paper Address?

The paper addresses the challenge of accurately translating spoken academic presentations, specifically from the ACL (Association for Computational Linguistics) conference, into multiple languages. This task is particularly difficult due to:

  • Technical Domain: Academic talks are filled with specialized terminology, acronyms, and concepts unique to the field of NLP.
  • Speaker Variability: Researchers come from diverse backgrounds, leading to a wide range of accents, which can significantly degrade Automatic Speech Recognition (ASR) performance.
  • Multilingualism: The goal is to translate into 10 different target languages, requiring robust multilingual models.
  • Speech Translation (ST): This is a cascaded task, meaning errors in the initial speech-to-text (ASR) stage can propagate and negatively impact the subsequent text-to-text (Machine Translation - MT) stage.

The authors aim to build high-quality ST systems that can improve the dissemination of scientific knowledge and make the field more inclusive.

Methods

The JHU team developed two main systems: one for the unconstrained subtrack (fewer restrictions on data and models) and one for the constrained subtrack (more limitations). Both systems are cascaded, consisting of an ASR component followed by an MT component.

Unconstrained System: * ASR: They utilized the Whisper model (Radford et al., 2022), known for its robustness to accents due to its massive pre-training dataset. To further improve performance on technical terms, they "prompted" Whisper with the abstracts of the talks, providing context. They also found that decoding the entire talk at once (rather than segment-by-segment) yielded better results. * MT: They experimented with large pre-trained multilingual models like NLLB-200, mBART50, and M2M100.

Constrained System: * ASR: They leveraged wav2vec 2.0 and HUBERT models, fine-tuning them in a two-stage process using TED-LIUM 3 and then the ACL 60-60 development data. A key innovation here was the use of an interpolated language model during decoding. This involved combining a general English LM with a domain-specific LM trained on ACL data and even individual paper abstracts to bias the ASR towards technical terms. * MT: They fine-tuned mBART50 and M2M100 models on the MuST-C dataset (English-to-X TED talks), hypothesizing its domain similarity. They explored both bilingual and multilingual fine-tuning.

Domain Adaptation: A significant effort was made to create and utilize in-domain data. They scraped ACL 2021 papers and talks to build a corpus for improving both ASR and MT. This involved extensive data filtering to match the format of the evaluation data.

Datasets

  • IWSLT 2023 Multilingual Task Data: Provided audio of 5 talks (approx. 1 hour total), transcripts, and multi-parallel translations into 10 languages. Also included paper abstracts.
  • ACL 2021 Proceedings and Workshops Data: Scraped by the authors, comprising ~1847 papers and ~1193 talks. This data was processed to create a domain-specific corpus for ASR and MT.
  • MuST-C: A dataset of professionally translated English TED talks, used for fine-tuning MT models in the constrained setting.
  • Standard ASR Corpora: Librispeech, VoxPopuli, Common Voice, TED-LIUM 3 were used for pre-training and fine-tuning ASR models.

Key Insights and Ideas

  • Whisper's Robustness: The Whisper ASR model proved effective for handling the diverse accents present in academic talks, even without extensive domain-specific fine-tuning.
  • Prompting for Context: Using paper abstracts as prompts for Whisper significantly improved technical term recall, demonstrating the power of providing in-domain context to large ASR models.
  • Interpolated LMs for ASR: For the constrained ASR, interpolating language models (general + domain-specific + abstract-specific) was crucial for improving both Word Error Rate (WER) and technical term recall.
  • Domain Data Value: While the authors created a substantial in-domain corpus, they found its direct use for data augmentation (via SeqKD) yielded only marginal improvements. They speculate its primary benefit might be as source context during inference rather than training.
  • Cascaded System Performance: The unconstrained system showed a relatively small drop in performance (-5.7 chrF) when using ASR outputs compared to oracle transcripts, suggesting the cascaded approach with strong components is viable. The constrained system had a larger drop (-12.8 chrF), indicating more room for improvement in error propagation mitigation.

Key Results

  • Unconstrained ASR: Whisper-large with abstract, conclusion, intro, and author name prompting achieved the best ASR performance with a WER of 6.4 and technical term recall of 0.854.
  • Constrained ASR: HUBERT-960h fine-tuned with an interpolated LM (including ACL anthology data) achieved the best WER of 18.5 and technical term recall of 0.473.
  • Unconstrained MT: NLLB-200-3.3B generally performed best across most language pairs, achieving a chrF of 65.4 for en-ar.
  • Constrained MT: Fine-tuning with MuST-C data provided a boost over base models for some pairs, but M2M100 without fine-tuning was often competitive or superior.
  • Speech Translation: The final ST results showed a chrF reduction of -5.7 for the unconstrained system and -12.8 for the constrained system when using ASR outputs versus oracle transcripts.

Limitations and Future Work

  • Domain Data Augmentation: The authors did not see significant benefits from using their scraped ACL data for data augmentation via sequence knowledge distillation. They suggest it might be more useful as source context during inference.
  • Error Propagation: The larger drop in performance for the constrained ST system highlights the challenge of error propagation from ASR to MT. Future work could explore techniques like mixing ASR outputs with gold source sentences during MT training or joint ASR-MT training.
  • Prompting Mechanism: The paper notes that a more detailed look at Whisper's prompting mechanism could help understand its behavior and potential for improvement.
  • ACL Corpus Release: The authors are releasing a supplemental ACL audio and text corpus to encourage further research in this area.

Summary updated

BibTeX
@inproceedings{xinyuan2023jhu,
  author = {Henry Li Xinyuan and Neha Verma and Bismarck Bamfo Odoom and Ujvala Pradeep and Matthew Wiesner and Sanjeev Khudanpur},
  title = {{JHU IWSLT 2023 Multilingual Speech Translation System Description}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure