The University of Edinburgh's Submission to the IWSLT21 Simultaneous Translation Task

Sukanta Sen, Ulrich Germann, Barry HaddowIWSLT 2021 · OpenReview

TL;DR

This paper presents a novel re-translation approach for simultaneous English-German translation, aiming to balance translation quality with low latency. By combining dynamic masking and language model scores, the authors achieve a superior trade-off between latency and accuracy, outperforming baseline methods.

What Problem Does This Paper Address?

Simultaneous Spoken Language Translation (SLT) aims to translate speech in real-time, without waiting for the speaker to finish. This is crucial for applications like live events, but it presents a significant challenge: how to maintain high translation quality while minimizing latency. Traditional machine translation systems require complete sentences, but in SLT, the system must handle incomplete utterances. This often leads to a trade-off where reducing latency can degrade translation quality. The IWSLT 2021 shared task specifically focused on evaluating simultaneous SLT systems across different latency regimes.

Methods

The core of the submission is a re-translation approach. Unlike streaming methods that incrementally build translations, re-translation involves re-translating the entire source prefix every time new input arrives. This allows the use of standard, optimized NMT engines with beam search. However, a key challenge is "flicker," where the translation of a prefix can change significantly as more input is received, potentially leading to incoherent output.

To mitigate flicker and improve quality, the authors introduce two main techniques:

  • Language Model (LM) Score for Meaningful Units (MU): The system tracks the LM score of the current token compared to the previous one. If the score increases, it suggests a potentially "meaningful unit" that can be translated. The system then reads more tokens before re-translating, effectively introducing a "wait-k" mechanism guided by linguistic coherence.
  • Dynamic Masking: This technique, inspired by prior work, stabilizes translations by comparing the translation of a current prefix with the translation of an extended prefix (achieved by appending an "UNK" token). The longest common prefix (LCP) between these two translations is identified as the stable part, and the rest is masked to prevent unstable output.

These techniques are integrated into a re-translation framework, where the agent alternates between reading input and writing translated output, with the goal of producing a stable, high-quality translation with minimal delay.

Datasets

The training data for the NMT models comprised a mix of high-quality English-German parallel data from:

  • WMT 2020
  • MuST-C.v2
  • TED corpus
  • OpenSubtitle

For development, the authors used a concatenation of IWSLT test sets from 2014 and 2015. The final evaluation was performed on the IWSLT 2018 test set and the tst-COMMON set.

Key Insights and Ideas

  • Re-translation is viable with careful stabilization: The paper demonstrates that the re-translation approach, often criticized for flicker, can be made competitive with streaming methods if flicker is effectively managed.
  • LM scores can guide translation points: Using LM scores to identify "meaningful units" provides a data-driven way to decide when to commit to a translation, improving coherence.
  • Dynamic masking is crucial for stability: The dynamic masking approach effectively identifies and stabilizes the most reliable parts of the translation, preventing disruptive changes.
  • Combining techniques yields the best results: The most significant finding is that the combination of LM-based meaningful unit detection and dynamic masking (the "lm+mask" approach) offers the best trade-off between latency and BLEU score.

Key Results

The paper reports that their combined "lm+mask" approach achieved the best latency-quality trade-off across different latency regimes (low, medium, high) on the MuST-C.v2 test set. Specifically, Table 3 shows that for the "High" latency regime, their "lm+mask" approach achieved a BLEU score of 33.3 with an Average Lagging (AL) of 14.8, outperforming the baseline and other individual techniques. Figure 3 (BLEU vs. AL plots) visually confirms that the "lm+mask" approach consistently provides higher BLEU scores for a given AL, especially at lower latency levels.

Limitations and Future Work

  • Flicker Suppression vs. Quality: While the paper successfully suppresses flicker in the committed output, they acknowledge that their approach might "glue together incompatible fragments" if re-translations introduce errors in the initial parts of the hypothesis.
  • Modified Prefix Pairs: An alternative stabilization method using modified prefix pairs (where the target prefix is appended to the source prefix) was investigated but did not yield good results and was not pursued further.
  • Future Directions: The paper suggests that further exploration of dynamic masking and LM-based techniques could lead to even better simultaneous translation systems. The authors also note that while their approach improves the DAL-BLEU trade-off, further work on addressing the shortcomings of AL as a latency metric is valuable.

Summary updated

BibTeX
@inproceedings{sen2021university,
  author = {Sukanta Sen and Ulrich Germann and Barry Haddow},
  title = {{The University of Edinburgh's Submission to the IWSLT21 Simultaneous Translation Task}},
  booktitle = {IWSLT},
  year = {2021},
  url = {https://openreview.net/forum?id=2GKAly8uz5Z},
}
Figure