TL;DR
This paper presents the ON-TRAC consortium's systems for the IWSLT 2021 shared tasks on low-resource speech translation and multilingual speech translation. They achieved state-of-the-art results in low-resource speech translation by leveraging self-supervised learning with wav2vec features for ASR, and explored a novel dual-decoder Transformer for end-to-end multilingual speech translation.
What Problem Does This Paper Address?
The paper tackles the challenge of speech translation, particularly in low-resource scenarios where large amounts of parallel data are scarce. Traditional speech translation systems often rely on a cascaded approach: first transcribing speech to text (ASR), then translating the text (MT). This can lead to error propagation. For multilingual scenarios, efficiently handling multiple source and target languages within a single model is also a significant hurdle. The IWSLT 2021 shared tasks provided a benchmark to evaluate these challenges, focusing on Swahili-to-English and Congolese Swahili-to-French for low-resource settings, and a broader multilingual task involving several European languages.
Methods
For the low-resource speech translation task, ON-TRAC employed a cascaded approach. The Automatic Speech Recognition (ASR) component utilized a hybrid HMM/DNN system. A key innovation here was the use of wav2vec 2.0 XLSR-53 features, which are derived from a multilingual model pretrained via self-supervised learning. This approach proved highly effective for low-resource languages like Swahili. The ASR output was then fed into a neural machine translation (NMT) system. For the NMT, they explored both standard LSTM-based models with Luong-style attention.
For the multilingual speech translation task, the consortium investigated a more ambitious end-to-end approach using a Dual-decoder Transformer. This model is designed to jointly transcribe and translate speech, with separate decoders for ASR and ST that interact with each other. This aims to mitigate error propagation and potentially improve efficiency. They experimented with different model capacities, finding that a smaller configuration (6 encoder layers, 3 decoder layers) performed better for the given dataset.
Datasets
The primary datasets used were from the IWSLT 2021 evaluation campaign.
For low-resource speech translation:
- Coastal Swahili (swa) to English (eng)
- Congolese Swahili (swc) to French (fra)
Training data for ASR included:
- 5k instances of Congolese Swahili speech (provided by IWSLT organizers).
- A subset of the ALFFA corpus (read speech and broadcast news).
- A subset of the IARPA Babel Swahili Language Pack.
Training data for NMT included:
- For swa-eng: OPUS datasets (CCAligned, MultiCCAligned, WikiMatrix, Wikimedia, XLEnt, ParaCrawl) and a 5k parallel swa-eng dataset from IWSLT-2021.
- For swc-fra: Parallel corpora provided by organizers and OPUS.
For the multilingual speech translation task:
- A subset of the Multilingual TEDx corpus, featuring four source languages (Spanish, French, Portuguese, Italian) and five target languages (including English).
Key Insights and Ideas
- Self-supervised learning (SSL) for ASR is a game-changer for low-resource languages: The use of wav2vec 2.0 XLSR-53 features significantly reduced Word Error Rates (WER) for Swahili languages, demonstrating the power of pre-trained models in data-scarce scenarios.
- End-to-end multilingual ST is promising but requires careful tuning: The dual-decoder Transformer showed potential for jointly handling multiple languages, but its performance was sensitive to model capacity, with smaller models performing better on the given dataset.
- Cascaded systems still competitive for low-resource ST: Despite the allure of end-to-end models, the cascaded approach with strong ASR and MT components achieved competitive results.
- The impact of ASR features on MT quality can be complex: Interestingly, for the swc-fra pair, the MT performance (BLEU) was higher when using MFCC features for ASR, even though wav2vec features yielded lower ASR WER. This suggests a nuanced relationship between ASR error types and MT robustness.
Key Results
- Low-resource ASR: Achieved the best results among participants for both Swahili languages, with WERs of 31.25% (swa-eng) and 36.75% (swc-fra) on the test sets using wav2vec 2.0 features.
- Low-resource MT: The NMT systems achieved BLEU scores of 12.9 for swa-eng and 9.1 for swc-fra.
- Multilingual ST: The Dual-decoder Transformer outperformed single-decoder baselines on most language pairs, indicating its effectiveness for many-to-many translation. Specific BLEU scores are detailed in Table 5, showing performance across various language pairs. For instance, on the dev sets, the Dual-decoder achieved scores like 11.56 for es-en and 3.88 for pt-en.
Limitations and Future Work
- Limited exploration of multilingual ST: The authors acknowledge that they did not employ specific strategies to handle language pairs with no training data, and the low results on such pairs highlight the need for further investigation.
- Unexplained MT performance discrepancy: The reason for the higher BLEU score with MFCC-based ASR compared to wav2vec-based ASR for swc-fra was not investigated due to time constraints but is planned for future work.
- Further optimization of dual-decoder Transformer: While promising, the dual-decoder Transformer's performance could likely be further improved with more extensive hyperparameter tuning and architectural exploration.
- Investigating the impact of ASR error types on MT: A deeper analysis of how different types of ASR errors affect the downstream MT performance could lead to more robust cascaded systems.
Summary updated