The Volctrans Neural Speech Translation System for IWSLT 2021

Chengqi Zhao, Zhicheng Liu, Jian Tong, Tao Wang, Mingxuan Wang, Rong Ye, Qianqian Dong, Jun Cao, Lei Li

Introduction

The Volctrans team from ByteDance AI Lab participated in the IWSLT 2021 competition, focusing on two primary tracks: Offline Speech Translation (ST) and Simultaneous Text-to-Text Translation. Their system demonstrates a significant advancement in end-to-end (E2E) models, which translate audio directly to text. Traditionally, cascaded systems—combining Automatic Speech Recognition (ASR) and Neural Machine Translation (NMT)—have been the standard. However, E2E models are now challenging this dominance by mitigating error propagation and leveraging large-scale multi-task training.

Methodology

Progressive Multi-task Learning (PMTL-ST)

To overcome the scarcity of paired speech-translation data, the team employed Progressive Multi-task Learning. This approach involves initializing the E2E ST model with weights from pre-trained ASR and MT models. The system is then jointly trained on ASR, MT, and ST tasks to share representations across modalities.

The Volctrans architecture featuring a multi-modal encoder and task-specific decoding for ASR and MT/ST.

Audio Representation and Simultaneous Strategies

The system incorporates "fbank2vec," a module designed to learn contextualized audio representations from log Mel-filterbank features, enhancing the model's ability to extract meaningful features from raw audio. For the simultaneous track, the team utilized a "wait-k" strategy combined with "multi-path training." Multi-path training allows a single model to adapt to various latency requirements during inference, eliminating the need for separate models for different speed levels.

Data Augmentation and Robustness

The team utilized extensive datasets including MuST-C, LibriSpeech, and WMT 2020. They heavily relied on Back-Translation (BT) and Knowledge Distillation (KD) to generate high-quality synthetic data. To handle the "noisy" nature of ASR transcripts (e.g., lack of punctuation), they introduced an <ASR> tag in the MT input, signaling the model to expect irregular text structures.

Key Results

The Volctrans system achieved remarkable performance gains. In the Offline ST track, the E2E ensemble reached 31.3 BLEU on the tst2021 blind test set, notably outperforming their own highly optimized cascaded ensemble (29.5 BLEU). This suggests that E2E models can be more robust to specific data distributions, such as the shorter segments found in subtitle-like test sets.

BLEU scores vs. Average Lagging for simultaneous translation on EN→DE and EN→JA tasks.

In the simultaneous translation track, the systems exceeded competition benchmarks by approximately 7 BLEU points across various latency regimes. As shown in the performance curves, the multi-path and ensemble strategies proved particularly effective in maintaining high translation quality even at low latencies.

Conclusion

The Volctrans system for IWSLT 2021 highlights the growing viability of end-to-end speech translation. By integrating multi-task learning, advanced audio encoding, and robust data augmentation, E2E models have reached a level where they can compete with and even surpass traditional cascaded systems in real-world scenarios.

Summary updated

BibTeX
@inproceedings{zhao2021volctrans,
  author = {Chengqi Zhao and Zhicheng Liu and Jian Tong and Tao Wang and Mingxuan Wang and Rong Ye and Qianqian Dong and Jun Cao and Lei Li},
  title = {{The Volctrans Neural Speech Translation System for IWSLT 2021}},
  booktitle = {IWSLT},
  year = {2021},
  doi = {10.18653/v1/2021.iwslt-1.6},
}
Figure