TL;DR
The Kyoto Speech-to-Speech Translation System for IWSLT 2023 presents a novel approach to speech-to-speech translation by combining a dual-decoder Transformer for speech-to-text translation with a cascaded acoustic model and vocoder for text-to-speech synthesis. This system achieves competitive results by leveraging multi-task learning and carefully filtering large datasets for improved training.
What Problem Does This Paper Address?
Speech-to-speech translation (S2ST) aims to directly translate spoken language from one language to another, spoken in the target language. This is a complex task that bridges the gap between automatic speech recognition (ASR), machine translation (MT), and text-to-speech synthesis (TTS). Traditional S2ST systems often rely on a pipeline of separate ASR, MT, and TTS models, which can lead to error propagation and a lack of end-to-end optimization. This paper addresses the challenge of building an effective S2ST system by proposing an integrated architecture that can learn these tasks jointly and efficiently, while also tackling the issue of data scarcity and quality in large-scale S2ST training.
Methods
The core of the Kyoto S2ST system is a dual-decoder Transformer for the speech-to-text (ST) component. This architecture is designed for multi-task learning, featuring two decoders: one for ASR and another for ST. These decoders can exchange information through an interactive attention mechanism, where sub-layers in one decoder can attend to the hidden states of the other. This allows for a more integrated learning process than separate ASR and MT models.
For text-to-speech synthesis, the system employs a cascade approach, utilizing a pre-trained FastSpeech 2 model as the acoustic model and a HiFi-GAN model as the vocoder. This is a standard but effective approach for generating high-quality speech from text.
Datasets
The paper utilizes two primary datasets for training the speech-to-text translation system:
- MuST-C v2: This dataset includes Chinese translations and is noted for its potentially higher translation quality due to human translation.
- GigaST: This dataset is larger and in the same domain as the development and test data but contains machine-generated translations, which may be of lower quality.
To address the quality concerns with GigaST, the authors implemented a filtering strategy. They used sentence embeddings from LASER and Sentence Transformers to calculate the cosine similarity between source and target sentences. The top 10% of data based on these similarity scores were selected. Table 1 provides the sizes of the datasets before and after filtering.
For text-to-speech synthesis, the system leverages pre-trained models trained on the AISHELL-3 dataset.
Key Insights and Ideas
- Interactive Attention for Multi-Task Learning: The dual-decoder Transformer with interactive attention is a key innovation, allowing the ASR and ST decoders to learn from each other, potentially improving the performance of both tasks simultaneously.
- Data Filtering is Crucial: The experiments highlight the significant impact of data filtering on performance. Filtering the large GigaST dataset using sentence embeddings (especially LASER) led to substantial improvements, suggesting that even with machine-generated translations, careful selection can yield better results.
- Dataset Domain vs. Quality Trade-off: The paper demonstrates that while domain-specific data (GigaST) might be beneficial, the quality of translations (MuST-C) also plays a critical role. The best results were achieved by training on the filtered GigaST dataset.
- LASER Outperforms Sentence Transformers for Filtering: The study found that LASER was more effective than Sentence Transformers in identifying high-quality sentence pairs for filtering the GigaST dataset.
Key Results
The paper reports experimental results in Table 2, showing BLEU scores for different training configurations.
- Training solely on GigaST filtered by LASER achieved a BLEU score of 13.96.
- Training on MuST-C yielded a BLEU score of 9.71.
- Using GigaST filtered by Sentence Transformers resulted in a BLEU score of 11.57.
- Transfer learning approaches (e.g., training on MuST-C then GigaST) did not show significant improvement over training solely on the filtered GigaST dataset.
The system's text-to-speech component, using FastSpeech 2 and HiFi-GAN, is noted to have produced high-quality audio, with only a small percentage of test set audio files failing to generate due to preprocessing issues.
Limitations and Future Work
Limitations:
- A small percentage (<1%) of audio files could not be generated due to grapheme-to-phoneme conversion issues where English words were converted to empty strings.
- The paper notes that while GigaST is in-domain, its machine-generated translations may not be as high-quality as human translations.
Future Work:
The authors plan to explore:
- Multi-level pre-training using SpeechUT with phonemes as units.
- Investigating Encodec-based speech synthesis for improved representation of emotions and vocal patterns.
Summary updated