FINDINGS OF THE IWSLT 2020 EVALUATION CAMPAIGN

Ebrahim Ansari, Amittai Axelrod, Nguyen Bach, Ondrej Bojar, Roldano Cattoni, Fahim Dalvi, Nadir Durrani, Marcello Federico, Christian Federmann, Jiatao Gu, Fei Huang, Kevin Knight, Xutai Ma, Ajay Nagesh, Matteo Negri, Jan Niehues, Juan Miguel Pino, Elizabeth Salesky, Xing Shi, Sebastian Stüker, Marco Turchi, Alexander H. Waibel, Changhan WangIWSLT 2020 · DOI · OpenReview

TL;DR

The IWSLT 2020 evaluation campaign brought together 30 teams to tackle six challenging spoken language translation tasks, from real-time simultaneous translation to handling disfluent speech. This year saw a continued rise in end-to-end approaches, with results indicating that these models are increasingly competitive with traditional cascaded systems, especially when participants leverage custom data segmentation.

What Problem Does This Paper Address?

The International Conference on Spoken Language Translation (IWSLT) aims to advance the field of spoken language translation (SLT) by organizing annual evaluation campaigns. These campaigns focus on specific, challenging areas within SLT, such as real-time translation, handling noisy or non-native speech, and translating conversational dialogue. IWSLT 2020 specifically addressed the growing interest in end-to-end SLT models and explored their performance against established cascaded systems across various real-world scenarios. The campaign also investigated the impact of data quality and segmentation on translation performance.

Methods

The IWSLT 2020 campaign featured six distinct tracks, each with its own goals and evaluation metrics:

  • Simultaneous Speech Translation: Focused on low-latency translation of talks, evaluating both speech-to-text and text-to-text translation. Systems were assessed on quality (BLEU, TER, METEOR) and latency (Average Proportion, Average Lagging).
  • Video Speech Translation: Explored multimodal translation by combining video and audio sources for translation into text.
  • Offline Speech Translation: Investigated English-to-German translation using either cascaded or end-to-end approaches, with a focus on performance on TED talks data.
  • Conversational Speech Translation: Aimed to translate disfluent Spanish speech or text into fluent English, exploring the challenges of disfluency removal and the use of fluent references.
  • Open Domain Translation: Focused on Chinese-Japanese translation, encouraging research on noisy parallel web corpora and data provenance.
  • Non-Native Speech Translation: Addressed the realistic scenario of translating non-native spontaneous speech in noisy conditions, with English as the source and Czech or German as targets.

Participants were encouraged to use provided datasets and were evaluated using standard metrics like BLEU, TER, and METEOR, with specific latency and ASR metrics for relevant tracks.

Datasets

The campaign utilized a variety of datasets, including:

  • MuST-C: A multilingual speech translation corpus used across several tracks.
  • TED Talks: A common source for offline and non-native speech translation tasks.
  • How2 dataset: Used for English-Russian translation in the video speech translation track.
  • Fisher Spanish speech (disfluent) dataset: The core dataset for the conversational speech translation task, augmented with new fluent translations.
  • Global Voices, News Commentary, Ubuntu, OpenSubtitles, TED talks, Wikipedia, Wiktionary, WikiMatrix, Tatoeba.org: These were compiled to form the "existing_parallel" dataset for the Open Domain Translation task.
  • Webcrawled data: A large, noisy dataset harvested for the Open Domain Translation task.
  • Antrecorp, Khan Academy, SAO: Datasets specifically curated for the Non-Native Speech Translation task, featuring non-native speakers and noisy conditions.

Key Insights and Ideas

  • End-to-End Models Gaining Ground: The campaign observed a steady increase in the adoption of end-to-end systems, particularly in offline and non-native speech translation, with some end-to-end systems achieving top scores.
  • Segmentation is Crucial: For tasks involving unsegmented or noisy input (like non-native speech), the ability to perform accurate input segmentation significantly impacts translation quality. Systems that developed their own segmentation strategies often outperformed those relying on provided, potentially suboptimal, segmentations.
  • Data Augmentation and Filtering Pay Off: Participants consistently benefited from data augmentation techniques (e.g., back-translation, noise injection) and careful data filtering, especially when dealing with noisy or out-of-domain web-crawled data.
  • Multimodality Shows Promise: The introduction of the Video Speech Translation track highlights the growing interest in leveraging multiple modalities (audio and video) for improved translation.
  • Non-Native Speech is Challenging: The Non-Native Speech Translation task underscored the significant impact of ASR quality on overall translation performance, especially in noisy and disfluent conditions.

Key Results

  • Simultaneous Speech Translation: The ON-TRAC system demonstrated competitive latency-quality trade-offs across various regimes in both text and speech tracks.
  • Offline Speech Translation: End-to-end systems achieved the highest BLEU scores on the test set, indicating they are now on par with or surpassing cascaded approaches, particularly when custom segmentation was employed.
  • Conversational Speech Translation: Systems that did not use fluent references in training showed promising results, suggesting generalization beyond explicit annotations is achievable. The IIT Bombay submission achieved the highest BLEU score when using ASR output as input.
  • Open Domain Translation: Data filtering proved highly effective, leading to significant performance improvements (2-5 BLEU points). Participants who invested in filtering and augmentation achieved top results.
  • Non-Native Speech Translation: The results showed considerable variation among systems, heavily influenced by ASR quality and segmentation. APPTEK/RWTH achieved the best ASR and translation scores for English-German.

Limitations and Future Work

  • Latency Metrics: For future editions of the simultaneous speech track, the inclusion of wall-clock time as part of the official latency metric is planned, requiring a more controlled evaluation environment.
  • Manual Evaluation: For tasks like non-native speech translation, where segmentation and ASR quality are critical, manual evaluation is acknowledged as desirable but challenging to conduct objectively.
  • Data Scarcity: For some tasks, particularly conversational speech translation, the dataset size was smaller, highlighting the need for effective techniques to handle data sparsity.
  • End-to-End vs. Cascade: While end-to-end systems are showing strong performance, the paper notes that their effectiveness can still depend on critical pre-processing steps like segmentation, suggesting further research is needed to fully understand the gap between the two paradigms.
  • New Challenges: The authors encourage researchers to propose new challenge areas for future IWSLT editions.

Summary updated

BibTeX
@inproceedings{ansari2020findings,
  author = {Ebrahim Ansari and Amittai Axelrod and Nguyen Bach and Ondrej Bojar and Roldano Cattoni and Fahim Dalvi and Nadir Durrani and Marcello Federico and Christian Federmann and Jiatao Gu and Fei Huang and Kevin Knight and Xutai Ma and Ajay Nagesh and Matteo Negri and Jan Niehues and Juan Miguel Pino and Elizabeth Salesky and Xing Shi and Sebastian Stüker and Marco Turchi and Alexander H. Waibel and Changhan Wang},
  title = {{FINDINGS OF THE IWSLT 2020 EVALUATION CAMPAIGN}},
  booktitle = {IWSLT},
  year = {2020},
  doi = {10.18653/v1/2020.iwslt-1.1},
}
Figure