TL;DR
This paper tackles the challenge of conversational speech translation for Estonian, a low-resource language, by finetuning large, publicly available end-to-end models. They demonstrate that synthetic data, generated from ASR training data, significantly boosts translation accuracy, making models like SeamlessM4T and Whisper competitive with traditional cascaded systems.
What Problem Does This Paper Address?
Spoken language translation (SLT) is crucial for bridging language barriers, especially for speakers of less common languages like Estonian. However, developing robust SLT systems for conversational speech is hampered by a scarcity of relevant training data. Existing large, pre-trained multilingual models, while powerful, often struggle with the nuances of spontaneous, conversational speech, particularly for low-resource languages. This paper aims to bridge this gap by exploring how to effectively finetune these powerful end-to-end SLT models for Estonian, using novel data generation techniques to overcome the data limitations.
Methods
The core of this research lies in finetuning existing end-to-end SLT models. The authors experimented with three prominent models: OpenAI's Whisper, OWSM 3.1 EBF, and Meta's SeamlessM4T.
- Data Generation: To address the lack of Estonian conversational speech translation data, they employed two primary methods:
- Web Scraping: They collected data from online sources like YouTube, TED talks, and news broadcasts, focusing on long-form conversational content with good quality subtitles.
- Synthetic Data Generation: They leveraged existing Estonian ASR training data and used machine translation (MT) to generate target language text. This approach was chosen over speech synthesis due to the difficulty in finding suitable MT corpora for conversational speech.
- Finetuning: The models were finetuned using this augmented dataset. Specific hyperparameters and training schedules were detailed for each model, including learning rate schedules, batch sizes, and optimizers. Whisper's ability to be finetuned for directions beyond its original English-only translation was a key aspect explored.
- Evaluation: The performance of both the original and finetuned models was evaluated using automatic metrics: BLEU and BLEURT. They also compared these end-to-end models against traditional cascaded systems (ASR + MT).
Datasets
- Training Data:
- Estonian ASR Data: TalTech Estonian Speech Dataset 1.0 (primarily broadcast sources, interviews, talk shows, parliamentary sessions).
- English ASR Data: Gigaspeech corpus (long-form recordings like audiobooks, podcasts, YouTube videos).
- Russian ASR Data: Russian TEDx talks and Deutsche Welle (DW) Russian YouTube channel.
- Web Scraped Data: ETV+, TED talks (with Estonian subtitles), TV7, and various YouTube channels.
- Evaluation Data: A custom evaluation dataset was compiled, featuring long conversational speech recordings from sources like press conferences, TV talk shows, YouTube videos, and broadcast news. The data was manually transcribed and translated by professional translators.
- Benchmarks: The paper references existing datasets like CoVOST 2 and FLEURS for comparison, noting that their custom evaluation data focuses on more challenging "in-the-wild" conversational speech.
Key Insights and Ideas
- Synthetic Data is Key: Finetuning with synthetic data generated from ASR corpora significantly boosts translation accuracy for low-resource languages like Estonian, even outperforming models trained on scraped web data.
- Whisper's Versatility: OpenAI's Whisper, originally trained for English translation, proves to be a highly effective base model that can be successfully finetuned for other translation directions (e.g., Estonian to Russian).
- SeamlessM4T's Robustness: SeamlessM4T demonstrates consistent performance across different directions and finetuning approaches, highlighting its robustness in handling conversational speech.
- Conversational Speech is Harder: End-to-end models, especially SeamlessM4T, show a noticeable drop in performance on conversational speech compared to read speech datasets like CoVOST 2 and FLEURS, underscoring the challenge.
- ASR Quality Matters: For Estonian source speech, using ASR instead of reference transcripts for MT input deteriorates BLEU scores, suggesting that ASR accuracy is a critical bottleneck.
Key Results
- Significant Accuracy Gains: Finetuning with synthetic data led to substantial improvements in translation accuracy for both Whisper and SeamlessM4T. For example, finetuned Whisper achieved a BLEU score of 33.2 for Estonian-English translation, a significant jump from its out-of-the-box score.
- Competitive Performance: The best finetuned models, particularly SeamlessM4T, achieved performance comparable to or even surpassing cascaded systems that use state-of-the-art ASR and MT models.
- Estonian-Russian Improvement: Finetuned SeamlessM4T showed a statistically significant improvement over the cascaded system and finetuned Whisper in the Estonian-Russian direction.
- Qualitative Usability: The authors suggest that their best finetuned models produce translations that are accurate and fluent enough for practical use in real-world scenarios, providing a YouTube link as an example.
Limitations and Future Work
- Subjective Evaluation: While automatic metrics were used, the authors note that subjective evaluation by the authors suggests that translations in the opposite direction (e.g., English to Estonian) have substantially lower quality.
- OWSM Performance: The OWSM 3.1 EBF model showed notably lower performance compared to Whisper and SeamlessM4T, even after finetuning.
- Future Work: The authors plan to explore simultaneous speech translation, where end-to-end models are particularly crucial.
Summary updated