TL;DR
This paper presents the IMS team's approach to the IWSLT 2021 Low-Resource Speech Translation task, focusing on Swahili-to-English and Swahili-to-French translation. They achieved top results by combining state-of-the-art cascaded ASR-MT systems with data augmentation and transfer learning, and also explored end-to-end models.
What Problem Does This Paper Address?
The paper tackles the challenge of speech translation for low-resource languages, specifically Swahili dialects (Coastal and Congolese) to English and French. Low-resource languages present a significant hurdle due to the scarcity of parallel data, making it difficult to train robust speech translation systems. The IWSLT 2021 task aims to advance research in this area by providing a benchmark for these under-represented languages.
Methods
The IMS team employed a two-pronged approach:
- Cascaded ASR-MT System (Primary Submission):
- ASR: Utilized a Conformer end-to-end ASR model with ESPnet implementation. They experimented with various data augmentation techniques (speed perturbation, SpecAugment) and pre-trained models (LibriSpeech, SPGISpeech, Russian Open STT, wav2vec 2.0 variants). A key innovation was the use of a multi-task framework for the MT system, which also incorporated an external language model.
- MT: Employed a Transformer big model. They explored three MT system variations:
- Vanilla NMT: Trained on standard parallel text data.
- PreprocNMT: Trained on ASR-like text generated from Swahili transcriptions.
- MultiNMT: A multi-task approach that jointly trained to translate ASR output (primary task) and regular Swahili text (auxiliary task). This approach proved most effective.
- End-to-End ST System (Contrastive): This system integrated the ASR encoder with the MT decoder, freezing most layers and fine-tuning only a few. This approach showed lower performance compared to the cascaded system.
Datasets
The paper utilized several datasets for training and evaluation:
- ASR:
- Shared task training data (Coastal Swahili and Congolese Swahili).
- Gamayun Swahili speech samples.
- IARPA Babel Swahili Language Pack.
- ALFFA dataset.
- Validation data was derived from the shared task training data and ALFFA.
- External LM training data included transcriptions from ASR training and ALFFA.
- MT:
- Shared task training data.
- Gamayun kit (English-Swahili and Congolese Swahili-French parallel corpora).
- Multiple corpora from the OPUS collection (e.g., ELRC_2922, GNOME, CCAligned, EUbookshop, Global Voices, JW300, ParaCrawl, MultiParaCrawl, Tanzil, TED2020, Ubuntu, WikiMatrix, wikimedia).
- Validation data consisted of 434 randomly sampled utterances per target language from the shared task training data.
- End-to-End ST:
- Shared task speech recordings with English or French transcriptions (obtained from the shared task or by running their MT system on Swahili transcriptions).
- External LMs for English and French were trained on 10M sentences from the OSCAR corpus.
Key Insights and Ideas
- Cascaded systems still dominate low-resource ST: Despite advancements in end-to-end models, the cascaded ASR-MT approach, when carefully tuned, achieved superior results for low-resource languages.
- Multi-task learning for MT is crucial: Training the MT system to handle both standard text and ASR-like output significantly improved its robustness and generalization, especially when dealing with imperfect ASR inputs.
- Pre-trained ASR models offer a strong starting point: Fine-tuning pre-trained ASR models, particularly those trained on large, diverse datasets (like Russian Open STT), provided a substantial boost in performance.
- Language-specific external LMs help: While an external LM improved end-to-end ST results, the optimal weights varied between languages, suggesting the need for language-specific tuning.
- Data augmentation and careful preprocessing are key: Techniques like SpecAugment for ASR and converting ASR output to a format suitable for MT were vital for improving performance.
Key Results
- The primary cascaded system achieved the best performance among all submitted systems for Congolese Swahili to English (BLEU 13.7) and Congolese Swahili to French (BLEU 7.7).
- It also secured the second-best result for Coastal Swahili to English (BLEU 14.9).
- The end-to-end system consistently performed about twice worse than the pipeline approach.
- The multi-task NMT system (multiNMT) showed significant improvements, reaching 20.07 BLEU for swa→eng when trained with ASR output.
- ASR validation WERs for the final system were 12.5% (swc) and 17.6% (swa).
Limitations and Future Work
- End-to-End ST Performance Gap: The end-to-end ST system did not match the cascaded systems' translation quality in low-resource settings. The authors acknowledge that there is still room for improvement in this area.
- Validation vs. Evaluation Score Gap: A large gap was observed between validation and evaluation scores for Coastal Swahili, suggesting potential bias towards the validation set in the ASR or MT models. The reason for this not occurring with Congolese Swahili is unclear.
- Future Work: The authors plan to further investigate end-to-end ST systems in future research.
Summary updated