TL;DR
This paper presents NAIST's submission to IWSLT 2023 for simultaneous speech translation (S2T and S2SS). They achieve strong results by leveraging pre-trained multilingual models, enhancing them with novel inter-connections and a bilingual prefix alignment technique for better prefix-to-prefix translation. For speech-to-speech, they cascade their S2T system with an incremental text-to-speech module.
What Problem Does This Paper Address?
Simultaneous Speech Translation (SimulST) aims to translate spoken language in real-time, presenting a significant challenge due to the need for low latency while maintaining translation quality. Traditional approaches often require specialized architectures, increasing complexity. This paper tackles the problem of building effective SimulST systems, particularly for multiple language pairs (English to German, Japanese, Chinese for S2T, and English to Japanese for S2SS), by adapting and enhancing existing end-to-end models. A key challenge is the "prefix-to-prefix" nature of SimulST, where the model must translate partial input to partial output, which can lead to issues like over-translation or hallucination if not handled carefully. Furthermore, extending this to speech-to-speech (S2SS) adds the complexity of generating natural-sounding speech in real-time.
Methods
The core of their approach for simultaneous speech-to-text (S2T) is an end-to-end multilingual model built upon large-scale pre-trained speech and text models. Specifically, they use HuBERT-Large as the speech encoder and mBART50 as the text decoder.
Key novelties include:
- Inter-connection: To better leverage the rich information within the intermediate layers of the pre-trained speech encoder, they introduce a "weighted integration" of these layers. This adds learnable parameters to bridge the encoder and decoder, stabilizing training.
- Bilingual Prefix Alignment (BPA): To mitigate over-translation issues inherent in prefix-to-prefix translation, they augment their training data with prefix-to-prefix pairs generated using BPA. This technique helps the model learn more robust translations for partial inputs.
- SimulST Policy (Local Agreement): They employ a "local agreement" policy, specifically LA-2, which verifies the stability of hypotheses across consecutive decoding steps to find the longest common prefix. This helps in making more reliable translation decisions.
- Incremental Text-to-Speech (TTS) for S2SS: For the speech-to-speech track, they build a cascaded system. The S2T output is fed into an incremental TTS module composed of three parts: a pronunciation estimation model, an acoustic model, and a neural vocoder. This module is designed to generate speech incrementally, maintaining low latency.
Datasets
The authors utilized several publicly available datasets for training and evaluation:
- MuST-C v2.0: Used for English-to-German (En-De), English-to-Japanese (En-Ja), and English-to-Chinese (En-Zh).
- CoVOST-2: Also used for En-De, En-Ja, and En-Zh.
- MuST-C v1.0, Europarl-ST, and TED-LIUM: Used for En-De.
- JSUT corpus: Used for training the Tacotron2 and Parallel WaveGAN models in the TTS component.
- Balanced Corpus of Contemporary Written Japanese (BCCWJ): Used for training the pronunciation estimation model.
They also mention using the development and test portions of CoVOST-2 and Europarl-ST in their training data. For evaluation, they used the tst-COMMON portion of MuST-C v2.0.
Key Insights and Ideas
- Pre-trained models are powerful, but need adaptation: The success of using HuBERT and mBART50 highlights the power of large pre-trained models. However, the paper demonstrates that simply connecting them isn't enough; novel techniques like "Inter-connection" are crucial for effective integration.
- Prefix alignment is key for robust SimulST: The "Bilingual Prefix Alignment" technique significantly improves translation quality by directly addressing the challenges of prefix-to-prefix translation, especially for language pairs with differing word orders.
- Data filtering is critical for specific language pairs: For En-Ja, filtering prefix pairs based on a maximum ratio (4,000) yielded the best results, suggesting that careful data curation is vital, particularly for languages with significant word order differences.
- Speech-to-speech is still a hard problem: While their cascaded S2SS system shows promise, the results indicate that achieving high ASR quality and natural TTS simultaneously remains a significant challenge, with room for improvement in both components.
- Latency vs. Quality Trade-offs are nuanced: The paper explores various latency metrics and shows that different policies (like local agreement vs. fixed-size segmentation) have different impacts on the quality-latency trade-off, especially when considering computation-aware latency.
Key Results
- Speech-to-Text (S2T): The fine-tuned models with Bilingual Prefix Alignment consistently outperformed the baseline offline model across all language pairs (En-De, En-Ja, En-Zh). For En-Ja, the best results were achieved with a specific data filtering ratio (4,000).
- Speech-to-Speech (S2SS): The S2SS system achieved a significantly lower ASR_BLEU score (nearly 5 points lower) compared to the S2T system, indicating challenges in the cascaded approach. The ASR_BLEU stagnated around 10.5 points, and the system had a character error rate of 28.3% relative to the S2T output.
- Latency: The paper presents detailed results on various latency metrics (AL, LAAL, ATD, etc.) showing how different chunk sizes and policies affect performance. For instance, fixed-size segmentation was found to be better than local agreement for computation-aware latency.
Limitations and Future Work
- S2SS Quality: The authors acknowledge that the speech-to-speech system still has significant room for improvement, particularly in ASR quality and the naturalness of synthesized speech.
- Inter-connection for En-Zh: While Inter-connection improved performance for En-De and En-Ja, it did not yield the same benefits for En-Zh. The authors attribute this to differences in speech features required for translation, suggesting that more language-specific adaptations might be needed.
- Future Work: The paper implicitly suggests further research into improving the S2SS pipeline, potentially by exploring end-to-end S2SS models or more advanced TTS integration. Investigating why Inter-connection performs differently across languages could also lead to more robust multilingual models. The exploration of computation-aware latency also opens avenues for more practical, real-world deployment.
Summary updated