Introduction
Speech Translation (ST) is a complex task that traditionally combines Automatic Speech Recognition (ASR) and Neural Machine Translation (NMT). In a pipeline approach, errors in the ASR stage—such as missing punctuation or incorrect casing—can significantly degrade the performance of the subsequent NMT stage. This problem is particularly acute in low-resource settings, such as Swahili-to-English translation, where high-quality parallel data is scarce.
The USYD-JD team addressed these challenges in the IWSLT 2021 Swahili-to-English task by developing a robust pipeline that maximizes the utility of limited data and enhances the NMT component's resilience to noisy ASR outputs. Their system achieved a state-of-the-art BLEU score of 25.3, securing the top spot in the competition.
System Architecture and Text Refinement
The core of the USYD-JD system is an $ASR \rightarrow NMT$ pipeline. To mitigate error propagation, the team focused on "rejuvenating" the text between the two stages.
ASR and Punctuation Correction
The system uses the Kaldi tool for ASR. Because raw ASR output often lacks proper casing and punctuation, the team integrated SlotRefine, a model designed to restore these elements. This step is crucial for the NMT model, which is typically trained on clean, punctuated text.
Advanced Pre-training
To improve the NMT model's fundamental understanding, the authors employed two pre-training strategies:
- Bidirectional Pre-training (BiPT): The model was trained to translate in both directions (Swahili $\leftrightarrow$ English), fostering a deeper bilingual representation.
- Denoising Pre-training (DPT): The model was trained to reconstruct clean text from intentionally "noised" versions (containing word swaps or deletions), making it more robust against ASR errors.
Data Augmentation and Self-Training
A key factor in the system's success was the massive expansion of training data through iterative self-training.
Bidirectional Self-Training
The team used an iterative process where "teacher" models generated synthetic data for "student" models. This process spanned multiple rounds, significantly increasing the volume of training data from the initial 2.4 million parallel sentences to over 50 million.
Smart Data Selection
Rather than using all available monolingual data for back-translation, the team used BERT-based scoring and Moore-Lewis filters to select "in-domain" data. This targeted approach provided a +1.9 BLEU boost over standard back-translation methods. They also utilized Tagged Back-Translation to help the model distinguish between authentic and machine-generated text.
Key Results and Insights
The combination of these techniques led to a cumulative gain of +10.8 BLEU points over a baseline Transformer ensemble.
- Final Performance: 25.3 BLEU on the official IWSLT test set (1st place).
- Ablation Studies: Bidirectional and Denoising pre-training contributed +1.7 BLEU, while iterative self-training added +2.5 BLEU.
- Reranking: The use of a MIRA ranker with Right-to-Left (R2L) and Target-to-Source (T2S) models further refined the final output.
Conclusion
The USYD-JD system demonstrates that a well-optimized pipeline, supported by sophisticated data augmentation and pre-training, can excel even in low-resource speech translation tasks. While the pipeline approach is effective, the authors note that error propagation remains a challenge. Future research will explore end-to-end models that incorporate cross-modal structures (audio and vision) to bypass the text-bottleneck and further improve robustness.
Summary updated