TL;DR
Samsung R&D Institute presents their system for the IWSLT 2020 End-to-End Speech Translation task, achieving a significant 3 BLEU improvement over their previous year's submission. They detail a combination of data augmentation, architectural tweaks like ResNet-like convolutional layers, and a dual-learning approach that boosts performance in low-resource speech translation settings.
What Problem Does This Paper Address?
The paper tackles the challenge of building high-quality end-to-end speech translation systems, particularly in scenarios with limited parallel training data, which is common for many language pairs. Traditional speech translation systems involve separate Automatic Speech Recognition (ASR) and Machine Translation (MT) components, but end-to-end systems aim to simplify this by directly translating speech to text in another language. The goal is to improve the accuracy and robustness of these end-to-end systems, making them more competitive with established pipeline approaches.
Methods
The authors built upon their previous year's Transformer-based architecture, introducing several key improvements:
- Dual Learning: They implemented a dual-learning setup, where a second decoder is added to perform ASR. This allows the system to be trained simultaneously on both speech-to-text and speech-to-translation tasks, sharing the encoder and convolutional layers. This approach yielded a notable BLEU increase.
- Spectrogram Augmentation: To increase data diversity and robustness, they employed several augmentation techniques:
- Warping: A technique that shifts parts of the spectrogram, simulating distortions like insertions and deletions.
- Spectrogram Noise: Adding multiplicative noise to the spectrogram.
- Randomized Step/Window Size: Varying the parameters of the spectrogram processing.
- ResNet-like Convolutional Layers: Instead of solely relying on more channels, they incorporated ResNet-like convolutional layers to process the audio input. This allows for deeper convolutional processing, improving feature extraction.
- Increased Encoder Depth: The number of encoder layers was increased from 6 to 12, a common strategy in NLP to enhance model capacity.
- Larger Embedding Size: The embedding dimension was increased to 512.
- Data Augmentation (Audio): They applied audio-level augmentations using Sox's effects (tempo, speed, echo) to further diversify the training data.
- Data Filtering: The training data was filtered to remove low-quality ASR outputs (WER > 75%) and also filtered using statistical dictionary-based methods for translation quality.
- Synthetic Data Generation: They generated synthetic translations for corpora like Must-C and TEDLIUM2 using pre-trained MT models to augment the limited parallel data.
Datasets
The following datasets were used for training and evaluation:
- Training Data:
- iwslt-corpus
- TEDLIUM2
- Must_C corpus
- Machine-translated Librispeech corpus
- Evaluation Data:
- tst2010
- tst2015
- tst2019
- tst2020
They also mention using WMT data (Paracrawl, Europarl, OpenSubtitles) to train the synthetic translation models.
Key Insights and Ideas
- Dual Learning Synergy: The dual-learning approach, combining ASR and SLT decoders, proved effective, showing a significant BLEU improvement even on non-augmented data. This suggests that jointly learning related tasks can boost performance.
- Augmentation Pays Off: Spectrogram augmentation techniques like warping and noise, along with audio-level augmentation, significantly contributed to the overall performance gains. This highlights the importance of robust data augmentation in low-resource settings.
- Deeper Convolutions are Beneficial: The adoption of ResNet-like convolutional layers for initial audio processing, allowing for deeper feature extraction, provided a notable improvement.
- Segmentation Matters: Their custom segmentation approach, producing longer segments, led to better performance compared to standard segmentation, indicating that the model can effectively handle longer audio chunks.
Key Results
The authors report a cumulative improvement of almost 3 BLEU points compared to their 2019 baseline model. Specific contributions include:
- Additional training data: +0.7 BLEU
- Spectrogram augmentation: +0.2 BLEU
- Encoder layer depth (12 layers): +1.0 BLEU
- ResNet-like convolutions: +0.5 BLEU
- Combining all techniques and increasing embedding size to 512 resulted in the overall ~3 BLEU improvement.
- Their final model achieved a BLEU score of 25.97 on the tst2015 test set (using their custom segmentation).
- They outscored the best IWSLT 2019 pipeline system on tst2019.
Limitations and Future Work
- Training Time: Training the final, larger model took almost a month, indicating a trade-off between performance and computational resources.
- Segmentation Optimization: While they optimized their custom segmentation process using tst2015, further exploration into optimal segment lengths for different models might be beneficial.
- Future Work: The paper doesn't explicitly detail future work, but the success of their combined approach suggests continued exploration of data augmentation, architectural improvements, and potentially larger models or more diverse training data. The effectiveness of dual learning also opens avenues for exploring other related tasks.
Summary updated