The USTC-NELSLIP Systems for Simultaneous Speech Translation Task at IWSLT 2021

Dan Liu, Mengge Du, Xiaoxi Li, Yuchen Hu, Lirong Dai

TL;DR

This paper introduces the Cross Attention Augmented Transducer (CAAT), a novel architecture for simultaneous speech and text translation that improves upon existing methods by jointly optimizing translation quality and latency. Their CAAT-based systems achieved significant BLEU score improvements over previous state-of-the-art systems at IWSLT 2021, demonstrating its effectiveness in both speech-to-text and text-to-text translation tasks.

What Problem Does This Paper Address?

Simultaneous translation, where translation begins before the source utterance is fully received, is challenging because it requires balancing translation accuracy with minimal delay. Existing approaches often fall into two categories: fixed policies (like "wait-k," which can lead to excessive latency) or flexible policies (which are harder to optimize). The authors aim to address this by developing a more effective and optimizable approach for simultaneous translation, particularly for tasks involving speech, where data scarcity can be a bottleneck. They also tackle the issue of integrating policy learning directly into the translation model for better performance.

Methods

The core contribution is the Cross Attention Augmented Transducer (CAAT) architecture. It extends the RNN-Transducer (RNN-T) model, which is typically used for speech recognition and has a monotonic constraint. CAAT removes this constraint, allowing for more flexible reordering, and augments the standard Attention Encoder-Decoder framework.

Key methodological innovations include:

  • CAAT Architecture: This model separates the source attention mechanism from the target history representation, allowing for a deeper coupling of source context and target history. This is crucial for tasks like machine translation where reordering is common.
  • Latency Loss Function: A novel latency loss function is introduced, designed to be "mergeable" and account for the expectation of latency across all possible expanded translation paths. This allows for fine-grained control over the latency-quality trade-off.
  • Streaming Encoder: For speech processing, they employ a streaming encoder that uses block processing with both local and infinite left context, enabling efficient processing of speech data while maintaining context.
  • Data Augmentation: To combat data scarcity in speech translation, they leverage techniques like self-training (generating target translations from ASR outputs) and speech synthesis (using TTS to create synthetic audio data).

Datasets

The paper utilizes several datasets for training and evaluation:

  • Speech Translation Corpora: MuST-C, Europarl, and CoVoST2 (speech, transcription, and translation).
  • Speech Recognition Corpora: LibriSpeech and TED-LIUM3 (speech and transcription).
  • Text Translation Corpora:
    • EN→DE: MuST-C, Europarl, Rapid-2019, WIT3-TED, Commoncrawl, WikiMatrix, Wikititles, and Paracrawl.
  • EN→JA: WIT3-TED, JESC, kftt, WikiMatrix, Wikititles, and Paracrawl.
  • Monolingual Corpora: Europarl-v10, News-crawl-2019, and News-crawl-2020 for English, German, and Japanese.

They also mention filtering and selecting subsets of these datasets, particularly Paracrawl for the EN→DE task, due to its large size.

Key Insights and Ideas

  • Joint Optimization is Key: The paper strongly suggests that jointly optimizing translation quality and latency within a single model (like CAAT) is more effective than using separate, fixed policies.
  • CAAT's Flexibility: By removing the monotonic constraint of RNN-T and augmenting attention, CAAT proves adept at handling the reordering inherent in machine translation, outperforming wait-k baselines.
  • Data Augmentation is Crucial for Speech: For speech translation tasks, where parallel data is often limited, data augmentation techniques like self-training and TTS synthesis provide substantial performance gains.
  • Cascaded Systems Offer Trade-offs: While end-to-end systems are promising, cascaded systems (ASR + MT) can still offer advantages in medium to high latency regions, though they inherently introduce more delay.

Key Results

  • Significant BLEU Improvements:
    • S2T: The proposed CAAT-based S2T system improved by an average of 11.3 BLEU points compared to last year's optimal systems.
  • T2T: The CAAT-based T2T system improved by an average of 4.6 BLEU points compared to last year's optimal systems.
  • Latency-Quality Trade-offs: CAAT consistently outperforms the wait-k baseline across various latency regimes, especially in low-latency scenarios.
  • Data Augmentation Impact: Adding data through self-training and speech synthesis boosted offline speech translation performance by nearly 7 and 1.5 BLEU points, respectively, on the MuST-C(v2) dataset.
  • Ensembling Benefits: Model ensembling further improved BLEU scores by roughly 0.5-1.5 points at different latency regimes for T2T translation.

Limitations and Future Work

  • Computational Cost: The CAAT model's joiner module has a higher computational complexity than conventional Transformers, which they mitigate by reducing the joiner's hidden dimension and using a larger decision step size.
  • Unsegmented Data: The paper notes that processing unsegmented data (like continuous speech streams without clear utterance boundaries) remains a challenge, and current methods might not be ideal. They suggest that more appropriate latency criteria might be needed for such data in the future.
  • Cascaded System Latency: While cascaded systems show promise, they inherently have higher latency than end-to-end systems due to the sequential nature of ASR followed by MT. Further work is needed to reduce this gap.
  • EN-JA Training Instability: The authors mention that the EN-JA translation task experienced some training instability, leading to a reduced joiner block parameter size and a more noticeable gap between CAAT and offline performance in that specific track.

Summary updated

Abstract

This paper describes USTC-NELSLIP's submissions to the IWSLT2021 Simultaneous Speech Translation task. We proposed a novel simultaneous translation model, Cross Attention Augmented Transducer (CAAT), which extends conventional RNN-T to sequence-to-sequence tasks without monotonic constraints, e.g., simultaneous translation. Experiments on speech-to-text (S2T) and text-to-text (T2T) simultaneous translation tasks shows CAAT achieves better quality-latency trade-offs compared to \textit{wait-k}, one of the previous state-of-the-art approaches. Based on CAAT architecture and data augmentation, we build S2T and T2T simultaneous translation systems in this evaluation campaign. Compared to last year's optimal systems, our S2T simultaneous translation system improves by an average of 11.3 BLEU for all latency regimes, and our T2T simultaneous translation system improves by an average of 4.6 BLEU.
BibTeX
@inproceedings{liu2021ustc,
  author = {Dan Liu and Mengge Du and Xiaoxi Li and Yuchen Hu and Lirong Dai},
  title = {{The USTC-NELSLIP Systems for Simultaneous Speech Translation Task at IWSLT 2021}},
  booktitle = {IWSLT},
  year = {2021},
  url = {https://openreview.net/forum?id=37tpyiV-v6t},
}
Figure