Without Further Ado: Direct and Simultaneous Speech Translation by AppTek in 2021

Parnia Bahar, Patrick Wilken, Mattia A. Di Gangi, Evgeny MatusovACL 2021

TL;DR

AppTek presents their 2021 IWSLT systems for both offline and simultaneous speech translation (ST). They introduce improvements in direct end-to-end ST and a novel posterior-tight integration method, alongside a flexible quality/latency trade-off mechanism for simultaneous ST, and a minimal-delay sentence segmentation technique.

What Problem Does This Paper Address?

Speech translation (ST) aims to convert spoken language from one language to another. This paper tackles two main challenges: offline ST, where the entire speech is translated after being fully processed, and simultaneous ST, where translation begins with minimal delay as the speech is being uttered. For offline ST, the challenge lies in achieving high translation quality, especially when dealing with the complexities of speech recognition and machine translation. For simultaneous ST, the core problem is balancing translation quality with latency – the delay between the spoken word and its translated output. This often involves difficult trade-offs, as reducing latency can sometimes degrade translation accuracy. Furthermore, handling unsegmented speech input (where sentence boundaries aren't explicitly marked) and ensuring smooth, natural-sounding translations are ongoing challenges.

Methods

AppTek's approach for offline ST involves two primary models:

  • Direct End-to-End ST: This model uses a Transformer architecture for both speech encoding and translation, benefiting from pre-training, data augmentation (synthetic data and SpecAugment), and domain adaptation.
  • Posterior Tight Integration: This method, inspired by previous work, collapses cascaded ASR and MT components into a single end-to-end trainable model. Instead of passing one-hot encoded words from ASR to MT, it passes ASR posteriors (probability distributions over words), allowing for joint optimization and potentially better performance.

For simultaneous ST, the system builds upon a previous alignment-based chunking approach. Key innovations include:

  • Flexible Quality/Latency Trade-off: They introduce a method to simulate different latencies during training by randomly dropping chunk boundaries with a probability ($P_{drop}$). This allows the model to learn to translate chunks of varying lengths, enabling a spectrum of quality-latency trade-offs at inference time.
  • Minimal-Delay Sentence Segmentation: To handle unsegmented ASR output, they developed a neural segmenter that uses a frame-level acoustic model to predict sentence boundaries. Crucially, it aims to minimize additional delay by making decisions based on a variable number of future words, allowing translation to proceed as soon as a sentence end is confidently predicted.
  • Streaming ASR: They utilize a hybrid LSTM/HMM model for the streaming ASR component, incorporating improvements like state carry-over between windows to enhance recognition quality and allow for smaller window sizes.

Datasets

The paper utilizes a variety of datasets for training and evaluation:

  • ASR Data: EuroParl, How2, MuST-C, TED-LIUM, LibriSpeech, Mozilla Common Voice, and IWSLT TED corpora (totaling approximately 2300 hours).
  • In-domain Data: MuST-C and IWSLT TED corpora were specifically chosen for domain adaptation.
  • Machine Translation Data: IWSLT TED, MUST-C, EuroParl ST, and CoVoST corpora were used for ST model training.
  • Synthetic Data: Generated by translating English ASR data with their MT model to augment parallel audio-translation pairs.
  • Evaluation Sets: MuST-C dev and IWSLT tst2014 were used as development sets. Official evaluation was performed on IWSLT 2021 test sets (tst2020 and tst2021).

Key Insights and Ideas

  • Posterior-based ST is competitive: Collapsing ASR and MT into a single model by passing ASR posteriors to the MT model shows strong performance, rivaling or even surpassing cascaded systems. This suggests a promising direction for end-to-end trainable cascaded-like systems.
  • Data augmentation is key for MT robustness: Applying techniques like synthetic data generation and SpecAugment significantly improves the MT model's robustness, particularly to varying chunk sizes in simultaneous translation.
  • Random chunk boundary dropping unlocks latency control: The novel approach of randomly dropping chunk boundaries during training effectively allows the model to learn a flexible range of quality-latency trade-offs for simultaneous translation, a significant improvement over previous fixed-point systems.
  • Minimal delay sentence segmentation is achievable: The proposed neural segmenter, by dynamically using future context and making early confidence-based decisions, successfully segments ASR output with minimal added latency, crucial for real-time applications.
  • ASR errors impact ST quality: Even with advanced ST models, recognition errors from the ASR system can lead to a noticeable drop in translation quality, highlighting the continued importance of robust ASR.

Key Results

  • Offline ST: AppTek's systems achieved competitive results, with their direct ST models showing improvements over previous years. The posterior model demonstrated strong performance, particularly when compared to cascaded systems.
  • Simultaneous ST: The systems provided a wide range of latency/quality trade-offs. For text-to-text, they achieved BLEU scores up to 33.1 with an Average Lagging (AL) of 12 words. For speech-to-text, they reached BLEU scores up to 21.0 with an AL of 4.0s. They observed a drop of about 5-6% in BLEU for speech-to-text compared to text-to-text at similar latencies, attributed to ASR errors.
  • Sentence Segmentation: The segmenter achieved a recall of 61.4% and a precision of 64.1% on the tst-COMMON test set, buffering words for only 0.4 positions on average.

Limitations and Future Work

  • Transformer depth in direct ST: The paper notes that for the direct ST model, using a deep Transformer encoder (12 layers) with a large decoder led to suboptimal solutions due to complex error propagation. Future work could explore optimal architecture depths.
  • ASR pretraining impact: While pretraining was beneficial for some ASR architectures, it degraded performance for the Transformer-based ASR model in direct ST. Further investigation into pretraining strategies for Transformer ST models is suggested.
  • Unsegmented input scoring: For simultaneous ST with unsegmented input, the paper acknowledges that official scoring conditions were not fully defined, and future work will involve analyzing scoring for this scenario more thoroughly.
  • MT model adaptation for segmentation: The paper mentions that fine-tuning on automatically segmented data improved robustness, but the segmentation of German reference translations using an edit distance variant was a complex process. Future work could refine this resegmentation process.
  • ASR errors: The inherent impact of ASR errors on ST quality remains a limitation, suggesting continued research into more robust ASR-ST integration.

Summary updated

BibTeX
@inproceedings{bahar2021without,
  author = {Parnia Bahar and Patrick Wilken and Mattia A. Di Gangi and Evgeny Matusov},
  title = {{Without Further Ado: Direct and Simultaneous Speech Translation by AppTek in 2021}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure