VUS at IWSLT 2021: A Finetuned Pipeline for Offline Speech Translation

Yong Rae Jo, Young Ki Moon, Minji Jung, Jungyoon Choi, Jihyung Moon, Won-Ik ChoIwslt 2021 · DOI

TL;DR

This paper presents a "finetuned" pipeline for offline English-to-German speech translation, achieving a BLEU score of 20.9. The authors emphasize a multi-stage approach involving careful data cleaning, ASR and MT module training, and sophisticated post-processing for sentence segmentation and merging, demonstrating the effectiveness of these engineering efforts.

What Problem Does This Paper Address?

Offline speech translation, the task of converting spoken language into text in another language without an internet connection, is a challenging problem. While end-to-end systems have gained traction, pipeline systems (speech-to-text, then text-to-text) offer better interpretability, allowing researchers to pinpoint where errors occur. However, pipeline systems are sensitive to the quality of intermediate text representations. This paper addresses the challenge of building a robust offline English-German speech translation system by focusing on improving each stage of a pipeline, particularly by addressing data quality and refining sentence-level processing.

Methods

The VUS system employs a cascaded pipeline architecture.

  • ASR Module: A wav2vec 2.0 base model is used for embeddings, augmented with a Transformer decoder. The training data was preprocessed by removing less useful speech samples based on Word Error Rate (WER) and cleansing erroneous text.
  • MT Module: A Transformer architecture is trained on the WMT 20 en-de news task dataset. Preprocessing included language identification and filtering sentences by length difference. A novel "written-to-spoken text conversion" step was implemented to normalize text, expand abbreviations, and convert digits into their spoken form.
  • Sentence Segmentation and Merging: To address fragmented ASR outputs, a DeepSegment model (BiLSTM-CRF) is used to merge ASR outputs. Sentence concatenation is further refined using Part-of-Speech (POS) information to ensure proper sentence boundaries.
  • Post-processing: The merged sentences are post-processed using POS tags to correct potential segmentation errors.

Datasets

  • ASR Training: Librispeech and MuST-C.
  • MT Training: WMT 20 en-de news task dataset.
  • Evaluation: IWSLT 2021 shared task dev and test sets. The paper also references the 2015 test set for ASR evaluation and the WMT dataset for MT evaluation.

Key Insights and Ideas

  • Pipeline Robustness through Data Curation: The paper strongly suggests that meticulous cleaning and filtering of training data, both for ASR (removing low-WER samples) and MT (script normalization, length filtering), significantly contributes to overall system performance.
  • The Power of Sentence-Level Processing: The authors highlight the importance of sophisticated sentence boundary detection and merging (DeepSegment, POS-based concatenation) to overcome the inherent fragmentation of ASR outputs, a crucial step for effective machine translation.
  • "Finetuned" as a Strategy: The authors clarify that "finetuned" in their context refers to a well-organized set of strategies and engineering efforts rather than a purely end-to-end approach, emphasizing the value of modular improvements.
  • Interpretability Advantage: The pipeline approach allows for easier diagnosis of errors, distinguishing between acoustic domain issues and translation challenges.

Key Results

  • The VUS system achieved a BLEU score of 20.37 on the dev set and 20.9 on the test set for English-German speech translation.
  • The ASR module achieved a WER of 28.3% on the 2015 test set.
  • The MT module recorded a BLEU score of 32.2 on the WMT dataset.
  • The authors observed that various pre- and post-processing steps led to meaningful performance gains.

Limitations and Future Work

  • Not Fully End-to-End: The system is a pipeline, not a fully end-to-end model, which the authors acknowledge by using the term "finetuned."
  • Constrained Condition: The use of NLTK for featurization in DeepSegment adheres to a constrained condition, implying potential for further exploration with unconstrained methods.
  • Future Work: The paper suggests that the transparent description of their pipeline system makes it "partially improvable," hinting at future work in refining individual components or exploring alternative architectures. The authors also mention the potential for further gains from pre- and post-processing techniques.

Summary updated

BibTeX
@inproceedings{jo2021vus,
  author = {Yong Rae Jo and Young Ki Moon and Minji Jung and Jungyoon Choi and Jihyung Moon and Won-Ik Cho},
  title = {{VUS at IWSLT 2021: A Finetuned Pipeline for Offline Speech Translation}},
  booktitle = {Iwslt},
  year = {2021},
  doi = {10.18653/v1/2021.iwslt-1.12},
}
Figure