Neural Simultaneous Speech Translation Using Alignment-Based Chunking

Patrick Wilken, Tamer Alkhouli, Evgeny Matusov, Pavel GolikACL 2020

TL;DR

This paper introduces a novel neural machine translation (NMT) approach for simultaneous speech translation that dynamically segments incoming speech into translatable chunks. By jointly training a chunk boundary detection component with the translation model and leveraging word alignments for chunk generation, they achieve significant improvements in translation quality over fixed-policy baselines.

What Problem Does This Paper Address?

Simultaneous machine translation (SMT) aims to translate speech or text in real-time, as it's being received. The core challenge lies in balancing translation latency with quality. If the system waits too long to receive more source input, latency increases, but translation quality can improve due to more context. Conversely, translating too early with limited context leads to lower quality. Existing methods often use fixed policies (e.g., waiting for a fixed number of words) or complex reinforcement learning approaches. This paper tackles the problem of creating a more effective and integrated approach to dynamic chunking for SMT, specifically addressing how to train models that can make intelligent, context-aware decisions about when to translate partial segments of speech.

Methods

The proposed method introduces a neural NMT architecture with two main components:

  • Source Chunk Boundary Detection: This component dynamically decides when to end a source chunk. It's implemented using a stacked bidirectional RNN encoder that takes word embeddings and a binary signal indicating a potential chunk boundary at the previous position. A softmax layer then predicts the probability of a chunk boundary at the current position. Crucially, this component is trained jointly with the translation model, unlike some prior work that used offline decisions.
  • Translation Model: This component translates the identified source chunks. It's based on a standard RNN attention model, extended with a stacked backward RNN encoder to incorporate context from both forward and backward directions within a chunk. The decoder also uses attention mechanisms to leverage the full source context seen so far.

A key innovation is the method for generating training data. Instead of relying on fixed policies or complex RL, they use alignment-based chunking. This involves:

  1. Generating unsupervised word alignments between source and target sentences.
  2. Extracting "minimal-length" monotonic phrase pairs (chunks) that respect alignment constraints.
  3. Optionally, improving these chunks by considering future source words (delayed chunking) and using language models to ensure contextually relevant words are included.

The training process is designed to be consistent with inference conditions, meaning the model learns to make decisions based on partial input.

Datasets

The experiments were conducted on the IWSLT 2020 English-to-German task. The authors utilized a combination of corpora:

  • In-domain: TED and MUST-C corpora.
  • Clean: News-Commentary, Europarl, and WikiTitles corpora.
  • Out-of-domain: OpenSubtitles, ParaCrawl, CommonCrawl, and rapid corpora (filtered for similarity).
  • Back-translated German: Additional data for training.

For speech recognition, they used approximately 2300 hours of transcribed English speech from corpora like MUST-C, TED-LIUM, and LibriSpeech.

Key Insights and Ideas

  • Alignment-based chunking is effective: Leveraging statistical word alignments to define chunk boundaries for training provides a more robust and context-aware segmentation than fixed policies.
  • Joint training is crucial: Training the chunk boundary detection and translation components together allows the model to learn how these decisions influence each other, leading to better overall performance.
  • Bidirectional encoders don't always outperform: While bidirectional encoders are often beneficial, in this context, they found that unidirectional encoders performed comparably, suggesting that for shorter chunks, the added complexity might not yield significant gains.
  • Delaying source chunk boundaries improves context: Shifting source chunk boundaries by a few words (D > 0) significantly improves translation quality by providing more context for decision-making.
  • Length normalization is important: Due to the nature of chunking, decoded chunk lengths can be shorter than in full sentence translation. Applying length normalization (specifically, a factor of 0.9) helps to align the model's output length with the expected target length.

Key Results

  • The proposed alignment-based chunking method outperformed a wait-k baseline by 2.6 to 3.7% BLEU absolute on the IWSLT 2020 English-to-German task for text input.
  • For speech input, the system also showed significant improvements over a wait-k baseline, achieving better BLEU scores and lower TER (Translation Error Rate).
  • The average lagging (AL) metric for simultaneous translation was reported to be around 4.1 to 4.5 seconds, indicating a good balance between latency and quality.
  • The study found that increasing the source boundary delay (D) improved AL but not consistently BLEU.
  • A length normalization factor of a = 0.9 was found to be optimal for tuning the output length.

Limitations and Future Work

  • Bidirectional Encoder Performance: The paper notes that bidirectional encoders did not consistently outperform unidirectional ones, suggesting that future work could explore architectures that better leverage bidirectional context for chunking.
  • Sensitivity to Thresholds: The model was found to be sensitive to the source boundary probability threshold (to), indicating that fine-tuning this parameter is important for optimal performance.
  • Future Work: The authors suggest exploring more sophisticated methods for generating training data and potentially incorporating more advanced language modeling techniques to further improve chunking decisions and translation quality. They also highlight the potential for applying this approach to other language pairs and domains.

Summary updated

BibTeX
@inproceedings{wilken2020neural,
  author = {Patrick Wilken and Tamer Alkhouli and Evgeny Matusov and Pavel Golik},
  title = {{Neural Simultaneous Speech Translation Using Alignment-Based Chunking}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure