TL;DR
This paper tackles the challenge of speech translation (ST) systems that rely on automatic speech recognition (ASR) outputs, which often contain errors. They propose using knowledge distillation (KD) to train a machine translation (MT) model to be more robust to these ASR errors by learning from a "teacher" model trained on clean transcripts. Their experiments show that combining KD with fine-tuning (FT) significantly improves translation quality, especially when KD is applied after FT.
What Problem Does This Paper Address?
Speech translation (ST) systems aim to convert spoken language in one language to text in another. Traditional "cascade" ST systems break this down into two steps: first, an Automatic Speech Recognition (ASR) system transcribes the speech, and then a Machine Translation (MT) system translates the transcription. A major hurdle for cascade ST is "error propagation": any mistakes made by the ASR system (e.g., misrecognizing words, incorrect punctuation) are passed on to the MT system, leading to degraded translation quality. While "end-to-end" ST systems aim to bypass this by translating speech directly, they often underperform cascade systems without extensive training. This paper focuses on improving cascade ST by making the MT component more resilient to ASR errors, specifically by training it on noisy, ASR-generated transcriptions.
Methods
The core of this paper's contribution lies in applying and combining two techniques to improve the robustness of the MT component in a cascade ST system:
- Knowledge Distillation (KD): This is a technique where a smaller "student" model learns from a larger, more capable "teacher" model. In this work, the teacher model is trained on clean, human-generated transcripts, while the student model is trained on ASR-generated, erroneous transcripts. The goal is for the student model to mimic the teacher's output, effectively learning to translate even when the input is imperfect. They specifically use "sequence-level KD," which considers the entire output sequence rather than just individual word probabilities.
- Fine-Tuning (FT): This is a standard technique where a pre-trained model is further trained on a specific task or dataset. Here, FT is used to adapt the MT model to the ASR-based input.
The authors explore several training strategies:
- MT<sub>clean</sub>: A baseline MT model trained only on clean transcripts.
- MT<sub>asr</sub>: A baseline MT model trained only on ASR-generated transcripts.
- MT<sub>asr</sub> + FT: The MT<sub>asr</sub> model fine-tuned.
- MT<sub>asr</sub> + KD: The MT<sub>asr</sub> model trained using knowledge distillation from MT<sub>clean</sub>.
- MT<sub>asr</sub> + FT + KD: Applying FT and KD simultaneously.
- KD → FT: First training with KD, then fine-tuning.
- FT → KD: First fine-tuning, then training with KD.
Datasets
The experiments were conducted on two datasets:
- MuST-C (English-Italian): A multilingual ST corpus derived from TED talks, providing approximately 250,000 segments of English speech, clean transcripts, and Italian translations. For training, ASR was trained on audio-transcript pairs, and MT was trained using clean transcripts as input and ASR outputs as noisy input.
- LDC Fisher Spanish Speech (Spanish-English): This dataset provided roughly 140,000 segments of Spanish speech, clean transcriptions, and erroneous transcriptions (ASR output), along with English fluent translations. The MT model used the fluent English translations as output.
Key Insights and Ideas
- ASR errors are a significant bottleneck: The MT model trained solely on ASR outputs (MT<sub>asr</sub>) performed considerably worse than one trained on clean transcripts (MT<sub>clean</sub>), highlighting the impact of ASR errors.
- Knowledge distillation helps bridge the gap: KD effectively transfers knowledge from a clean-transcript-trained model to an erroneous-transcript-trained model, improving robustness against ASR errors.
- Fine-tuning alone can be insufficient: While FT improved MT<sub>asr</sub>, it didn't fully recover the performance lost due to ASR errors, especially when the ASR errors were substantial.
- The order of FT and KD matters: Applying KD after FT (FT → KD) yielded the best results. This suggests that FT helps the student model learn good initial parameters, and KD then guides it towards the optimal performance ceiling set by the teacher model.
- FT can correct specific ASR errors: In some cases, FT was able to correct specific misrecognitions by the ASR system, leading to more accurate translations.
Key Results
- English-Italian: The best performing cascade system was MT<sub>asr</sub> + FT → KD, achieving a BLEU score of 30.2 on the ASR-based test set and 23.5 on the clean test set. This significantly outperformed the baseline MT<sub>asr</sub> (27.2 and 22.5, respectively) and even MT<sub>clean</sub> (29.7).
- Spanish-English: Similar trends were observed. The MT<sub>asr</sub> + FT → KD system achieved the highest BLEU scores, outperforming other configurations. The paper notes that the gap between ASR-based and clean test data was larger for Spanish-English due to higher ASR error rates (WER 36.5 vs. 14.49 for English-Italian).
Limitations and Future Work
- Focus on Cascade ST: The current work focuses on improving cascade ST systems. The authors express interest in incorporating these findings into end-to-end ST systems in future work.
- Implicit Error Correction: While KD and FT improved robustness, they did not directly address the ASR errors themselves. The paper notes that KD forces the student model to mimic literal teacher translations, which can sometimes include errors but also helps ignore disfluencies and produce more accurate translations by learning from the teacher's implicit error correction.
- Reproducibility: The paper provides detailed experimental setups and results, facilitating reproducibility.
Summary updated