Consistency is Key: On Data-Efficient Modality Transfer in Speech Translation

Hojin Lee, Changmin Lee, seung-won hwangEMNLP 2023 · OpenReview

Abstract

End-to-end approaches have shown promising results for speech translation (ST), but they suffer from data scarcity compared to machine translation (MT). To address this, progressive training has become a common practice, using external MT data during the fine-tuning phase. Despite its prevalence and computational overhead, its validity is not extensively corroborated. This paper conducts an empirical investigation and finds that progressive training is often ineffective. We identify a learning-forgetting trade-off as a critical obstacle and verify that consistency learning (CL) breaks this dilemma. The proposed method, combining knowledge distillation (KD) and CL, outperforms previous methods on the MuST-C dataset without additional data. Our proposed consistency-informed KD (cKD) achieves additional improvements.

Introduction

Speech Translation (ST) is the task of converting spoken audio in one language directly into text in another. It is a "data-hungry" field; while large-scale text-to-text Machine Translation (MT) data is available, high-quality audio-to-text (ST) data is scarce. To bridge this gap, researchers typically use progressive training: starting with an MT model and gradually feeding it more text data while fine-tuning it for speech. However, this creates a "knot" of a dilemma. If too much MT data is added, it interferes with the speech task; if too little is added, the model "forgets" the linguistic knowledge it learned from the text.

Methodology

We propose a shift from adding more data to ensuring internal consistency. Our approach relies on two main pillars:

Consistency Learning (CL)

We utilize R-Drop, a technique where the same input is passed through the model twice with different dropout masks. The model is then penalized if the two outputs aren't consistent. This acts as a powerful regularizer that prevents "catastrophic forgetting" of MT knowledge without needing to constantly replay old MT data.

Consistency-informed Knowledge Distillation (cKD)

Standard Knowledge Distillation (KD) forces a "student" ST model to mimic a "teacher" MT model. However, if the teacher is unsure or inconsistent about a specific word, it provides a noisy signal. Our cKD method weights the distillation loss: if the teacher's internal consistency is low for a specific token, the student ignores it.

Key Insights and Results

The most surprising finding is that progressive training—despite its prevalence—often yields negligible gains. In many cases, adding external MT data during fine-tuning actually hurt performance compared to our consistency-based approach.

MT BLEU scores over training steps showing the impact of Consistency Learning (CL) on preventing forgetting.

As shown in the analysis of MT BLEU scores over training steps, models trained without CL saw their MT performance plummet as they learned ST. Models with CL maintained high MT accuracy, proving that internal consistency is a viable substitute for massive data replay.

Performance Evaluation

We evaluated our models on the MuST-C dataset (English to German, Spanish, and French). Our best model (Base+cKD+CL) achieved an average BLEU score of 33.70 across three language pairs, outperforming previous high-end models like ConST and WACO.

Comparison of our proposed models against existing state-of-the-art speech translation methods on the MuST-C dataset.

Our method achieves results comparable to "Joint Pre-training" (JPT) methods but uses a fraction of the data and compute. While some models require 60,000 hours of speech data, our method performs excellently with just 960 hours. Furthermore, the improvements from cKD over standard KD were statistically significant ($p < 0.05$), validating that adaptive weighting based on consistency is a superior way to handle distillation.

Conclusion

This work demonstrates that internal consistency is key to efficient modality transfer in ST. While data-efficient, Consistency Learning (specifically R-Drop) can be slow to converge. Future work will explore integrating massive-scale multilingual MT models to further boost performance for low-resource languages.

Code and models are available at https://github.com/hjlee1371/consistency-s2tt.

Summary updated

Abstract

End-to-end approaches have shown promising results for speech translation (ST), but they suffer from its data scarcity compared to machine translation (MT). To address this, progressive training has become a common practice, of using external MT data during the fine-tuning phase. Despite of its prevalence and computational overhead, its validity is not extensively corroborated yet. This paper conducts an empirical investigation and finds that progressive training is ineffective. We identify learning-forgetting trade-off as a critical obstacle, then hypothesize and verify that consistency learning (CL) breaks the dilemma of learning-forgetting. The proposed method, which combines knowledge distillation (KD) and CL, outperforms the previous methods on MuST-C dataset even without additional data, and our proposed consistency-informed KD achieves additional improvements against KD+CL. Code and models are availble at https://github.com/hjlee1371/consistency-s2tt.
BibTeX
@inproceedings{lee2023consistency,
  author = {Hojin Lee and Changmin Lee and seung-won hwang},
  title = {{Consistency is Key: On Data-Efficient Modality Transfer in Speech Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=8zQ77tPTMR},
}
Figure