The AFRL IWSLT 2020 Systems: Work-From-Home Edition

Brian Ore, Eric Hansen, Tim Anderson, Jeremy GwinnupACL 2020

TL;DR

This paper details the Air Force Research Laboratory's (AFRL) submission to the IWSLT 2020 offline spoken language translation (SLT) task. They employed a cascaded system approach, breaking down the SLT pipeline into distinct modules for speech activity detection, automatic speech recognition, sentence segmentation, and machine translation, achieving competitive results.

What Problem Does This Paper Address?

The core challenge addressed is building an effective offline spoken language translation system for English-to-German TED Talks. This involves accurately transcribing spoken audio, segmenting it into meaningful units, and then translating those units into German. The "Work-From-Home Edition" context also implies potential constraints on resources and data access, which the authors navigated. The task specifically focuses on generating cased and punctuated German translations, adding complexity beyond simple word-for-word translation.

Methods

The AFRL team adopted a cascaded approach, meaning they built and integrated separate, specialized systems for each stage of the SLT pipeline:

  • Speech Activity Detection (SAD): A fully-connected neural network was used to identify segments of speech within the audio.
  • Automatic Speech Recognition (ASR): This involved a Kaldi-based system using a factorized time delay neural network (TDNN) with residual connections. They employed an interpolated bigram language model and a recurrent neural network (RNN) language model for rescoring.
  • Sentence Segmentation: A bidirectional RNN with an attention mechanism was used.
  • Machine Translation (MT): Two transformer-based MT systems were trained: one using the OpenNMT toolkit and another using the Marian toolkit.

A key methodological decision was to explore both cased/punctuated and lowercase/non-punctuated inputs for the MT systems, investigating how ASR output quality impacts translation. They also experimented with different approaches to sentence segmentation, including using only SAD-derived boundaries versus incorporating an automatic punctuator.

Datasets

  • Training Data:
    • TEDLIUM-v3: Used for ASR acoustic models and for initial language model estimation.
  • News Crawl and News Discussions: Used for language model estimation.
  • WMT18 news-translation task data (preprocessed): Formed the basis of the MT training corpus.
  • English TED data (4M words): Used for training the automatic punctuator.
  • Evaluation Data:
    • IWSLT 2020 evaluation campaign test sets (tst2014, tst2015, tst2018, tst2019, tst2020).
  • WMT news translation task test sets (newstest2018, newstest2019) were used for intermediate MT evaluation.

Key Insights and Ideas

  • Cascaded vs. End-to-End: The paper firmly sticks to a cascaded approach, suggesting it offers modularity and allows for optimization of individual components, which is a common strategy in SLT.
  • Impact of ASR Output on MT: The study highlights that for machine translation, using lowercase and non-punctuated ASR output generally yielded better results than cased and punctuated output, even when the MT model was trained on cased/punctuated data. This suggests that noise or errors introduced by the ASR in punctuation/casing can be detrimental.
  • Sentence Segmentation Nuances: While automatic segmentation is necessary for an end-to-end system, the paper found that defining additional sentence boundaries using an automatic punctuator sometimes led to a decrease in BLEU scores for Marian, but an improvement for OpenNMT. This indicates that the effectiveness of punctuation-based segmentation is model-dependent.
  • Sentence Length Matters: Analysis revealed that translation performance, particularly for the OpenNMT system, significantly degrades on sentences longer than 50 words. This is a crucial insight for understanding model limitations and potential areas for improvement.

Key Results

  • Primary Submission: The AFRL's primary submission achieved BLEU scores of 21.28 on tst2019 and 23.33 on tst2020.
  • MT System Performance:
    • OpenNMT systems trained on lowercase, non-punctuated data generally outperformed those trained on cased, punctuated data for the SLT task.
  • For the WMT news translation task (a proxy for MT quality), OpenNMT achieved scores up to 43.11 (Table 1) and Marian achieved scores up to 26.58 (Table 3, column A).
  • Automatic Segmentation Impact:
    • Defining sentence boundaries solely by SAD led to a decrease in BLEU for both Marian and OpenNMT compared to manually segmented data.
  • Adding an automatic punctuator improved OpenNMT's performance slightly but worsened Marian's.
  • Sentence Length Analysis: BLEU scores dropped substantially for sentences longer than 50 words, especially for OpenNMT.

Limitations and Future Work

  • Work-From-Home Constraints: The authors mention that limited computational resources due to working from home prevented them from fully exploring features like specifying pause durations in the punctuator or investigating alternative acoustic features for segmentation.
  • Punctuator Impact: The inconsistent impact of the automatic punctuator on different MT systems suggests further research is needed to understand its interaction with MT models.
  • Long Sentences: The significant performance drop on long sentences is a clear area for future work, potentially involving techniques to better handle longer sequences or more robust segmentation for them.
  • Resegmentation: The authors suggest resegmenting the MT training data to better align with ASR segmentation, which could be a promising avenue if automated efficiently.
  • Alternative Features: Exploring other acoustic features beyond pauses for sentence boundary identification is also proposed.

Summary updated

BibTeX
@inproceedings{ore2020afrl,
  author = {Brian Ore and Eric Hansen and Tim Anderson and Jeremy Gwinnup},
  title = {{The AFRL IWSLT 2020 Systems: Work-From-Home Edition}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure