Improving End-to-End Speech Translation by Imitation-Based Knowledge Distillation with Synthetic Transcripts

Rebekka Hubert, Artem Sokolov, Stefan Riezler

Abstract

End-to-end Automatic Speech Translation (AST) traditionally relies on high-quality "triplet" datasets consisting of audio, manual transcripts, and target translations. However, manual transcripts are costly and scarce. This paper introduces SynthIKD (Synthetic Imitation Knowledge Distillation), a framework that leverages imitation learning and synthetic transcripts generated by Automatic Speech Recognition (ASR) models. By matching the probability distributions of a Neural Machine Translation (NMT) teacher, the proposed method achieves significant performance gains, demonstrating that manual transcripts are not a prerequisite for effective knowledge distillation in AST.

Introduction

Automatic Speech Translation (AST) aims to map acoustic signals directly to text in a target language. While end-to-end models offer simplicity and reduced error propagation compared to cascaded systems, they are constrained by the need for parallel data triplets. The manual transcription of audio is a significant bottleneck, particularly for low-resource languages.

Knowledge Distillation (KD) has emerged as a solution, where a teacher NMT model guides an AST student. However, standard KD often assumes the teacher has access to ground-truth transcripts. This research investigates whether noisy, machine-generated transcripts can substitute for manual ones when combined with imitation learning (IL) techniques to address exposure bias.

Methodology: The SynthIKD Framework

The authors propose SynthIKD, which integrates three core components:

1. Imitation Learning (IL)

Standard training often suffers from "exposure bias," where a model fails during inference because it encounters its own previous errors—states it never saw during training. SynthIKD utilizes IL algorithms like Dagger and AggreVaTe. In this setup, the student model generates a sequence, and the teacher provides corrections for the student's specific mistakes, teaching the model how to recover from errors.

2. Synthetic Transcripts

To bypass the need for manual transcripts, the framework uses a frozen ASR model to generate synthetic transcripts from the source audio. These transcripts serve as the input for the NMT teacher.

3. Probability Distribution Matching (IKD+)

Rather than having the student predict only the teacher's most likely next word (hard labels), the IKD+ variant requires the student to match the teacher’s entire probability distribution over the vocabulary (soft labels). This provides a richer signal, helping the student understand the relationships between different word choices.

Experimental Setup

The researchers evaluated SynthIKD on three English-to-German benchmarks:

  • CoVoST-2: Approximately 430 hours of speech.
  • MuST-C: Approximately 408 hours of speech.
  • Europarl-ST: A smaller dataset (83 hours) used for additional validation.

The experiments compared Transformer-based and RNN-based student architectures against standard end-to-end baselines.

Key Results and Insights

Performance Gains

The SynthIKD+ approach yielded substantial improvements, outperforming baselines by approximately 4 BLEU points on both CoVoST-2 and MuST-C. This indicates that the combination of imitation learning and soft-label distillation is highly effective for AST.

Robustness of the NMT Teacher

A critical finding was the NMT teacher's ability to act as an error-corrector. Despite noise in the synthetic ASR transcripts, the teacher utilized its internal language model to provide accurate target translations. For instance, even when an ASR model misidentified a word (e.g., "plywood" instead of "plundered it"), the NMT teacher often inferred the correct meaning from the surrounding context.

Synthetic vs. Gold Transcripts

For Transformer-based models, the performance gap between using manual (gold) transcripts and synthetic transcripts was statistically insignificant. This suggests that the high cost of manual transcription can be avoided without sacrificing model quality, provided a sufficiently robust distillation framework is used.

Algorithm and Architecture Comparisons

  • Dagger vs. AggreVaTe: While AggreVaTe is theoretically more sophisticated, the simpler Dagger algorithm provided comparable results at a lower computational cost.
  • Architecture Sensitivity: Transformer students benefited significantly more than RNN students. The authors hypothesize that RNNs produced hypotheses too divergent from the truth for the teacher's single-step corrections to be effective.

Conclusion

This work demonstrates that imitation-based knowledge distillation can successfully leverage synthetic transcripts to train high-performing AST models. By focusing on error recovery and probability distribution matching, SynthIKD+ closes the gap between models trained with expensive manual transcripts and those trained with automated alternatives. Future research may extend these findings to more diverse language pairs and larger-scale pre-trained models.

Summary updated

BibTeX
@inproceedings{hubert2023improving,
  author = {Rebekka Hubert and Artem Sokolov and Stefan Riezler},
  title = {{Improving End-to-End Speech Translation by Imitation-Based Knowledge Distillation with Synthetic Transcripts}},
  booktitle = {IWSLT @ ACL},
  year = {2023},
  doi = {10.18653/v1/2023.iwslt-1.4},
}
Figure