TL;DR
This paper presents a cascaded speech translation system for Maltese, a low-resource language, for the IWSLT 2024 shared task. Despite using standard components like DeepSpeech for ASR and an LSTM for MT, the system achieved a BLEU score of 0.5, highlighting the significant challenges of low-resource ST, particularly with limited training data.
What Problem Does This Paper Address?
The paper tackles the challenge of Speech Translation (ST) for low-resource languages, specifically Maltese, within a constrained setting. Low-resource languages, like Maltese, have limited availability of speech technology resources, making it difficult to build effective ST systems. The "constrained" setting further exacerbates this by limiting the amount of training data available. Traditional ST approaches often rely on large parallel corpora, which are scarce for languages like Maltese. This work aims to build a functional ST system for Maltese under these severe data limitations, exploring a pipeline approach to leverage existing tools.
Methods
The authors employed a cascaded approach, breaking down the ST task into three sequential stages:
- Automatic Speech Recognition (ASR): They experimented with two ASR models:
- Hidden Markov Models (HMMs): A traditional approach to ASR, trained on 6 hours and 39 minutes of Maltese audio.
- DeepSpeech 1: A deep learning-based ASR system, trained on a combined dataset of approximately 12 hours of Maltese audio from the MASRI and CV datasets. This model was chosen over HMMs due to its superior performance.
- Language Modeling: To improve the ASR transcriptions, a KenLM probabilistic 3-gram model was trained on the "Korpuss Malti v4.0 Shuffled" dataset. This model was used to optimize the ASR output by selecting the most probable word sequences.
- Machine Translation (MT): For the translation from Maltese to English, three MT architectures were explored using the Fairseq library:
- LSTM: A single-layer bidirectional encoder-decoder model.
- Transformer (Base): A standard Transformer architecture with 6 encoder/decoder layers.
- Transformer (Large): A larger Transformer architecture with more dimensions.
The LSTM model was ultimately selected for the final pipeline due to its best performance on the development set.
Datasets
- MASRI dataset: Used for training the HMM and DeepSpeech ASR models.
- CV dataset: Used in conjunction with MASRI for training the DeepSpeech ASR model.
- Korpuss Malti v4.0 Shuffled: Used for training the KenLM language model. This dataset is referenced by the shared task organizers.
- IWSLT 2024 Shared Task Datasets: The system was evaluated on the official test sets for the constrained task (CV and Masri).
Key Insights and Ideas
- Low-resource ST is a significant challenge: Even with standard, well-established components, achieving high performance in low-resource settings with constrained data is extremely difficult.
- Cascaded systems are sensitive to ASR errors: The poor performance of the ASR system directly impacted the MT system, as errors in transcription were propagated and often treated as unknown tokens by the MT model.
- Data scarcity limits model effectiveness: The paper repeatedly emphasizes that the limited training data is the primary bottleneck, preventing models from learning robust representations and leading to poor performance.
- LSTM's potential in low-resource settings: While Transformers are generally favored, the paper hypothesizes and finds that LSTMs can still be competitive in low-resource scenarios due to their potentially lower data requirements compared to large Transformer models.
Key Results
- ASR Performance: The DeepSpeech ASR model achieved a Word Error Rate (WER) of 97.15% on the overall test set. The HMM model performed worse with a WER of 112.33%.
- MT Performance (Pipeline): The final cascaded system achieved an overall BLEU score of 0.5.
- CV Test Set: BLEU 0.6, CHRF-2 15.79
- Masri Test Set: BLEU 0.2, CHRF-2 14.50
- Impact of KenLM: Using the KenLM model provided some improvement in the pipeline's BLEU score (0.52 for CV, 0.21 for Masri with KenLM vs. 0.48 for CV, 0.23 for Masri without KenLM), but it was not enough to overcome the fundamental ASR limitations.
- MT Architecture Comparison (Development Set):
- LSTM: BLEU 25.76, CHRF-2 44.57
- Transformer (Base): BLEU 24.54, CHRF-2 44.15
- Transformer (Large): BLEU 25.20, CHRF-2 43.57
Limitations and Future Work
- Limitations:
- Severe data scarcity: The primary limitation is the lack of sufficient training data for both ASR and MT, which severely hampers performance.
- ASR errors propagate: The cascaded nature of the system means that errors from the ASR stage significantly degrade the MT output.
- "Out of domain" issues: The MT system struggled with incoherent or out-of-domain speech recognition outputs.
- Named entities and phonetic misspellings: The ASR system had particular difficulty with named entities and phonetic variations, leading to incorrect transcriptions.
- Future Work:
- Higher-quality training data: Acquiring more and better-quality training data is crucial.
- Improved ASR: Future work should focus on enhancing ASR performance, including better handling of named entities and phonetic variations.
- Pre-trained models: Investigating the use of pre-trained models (e.g., for ASR) could help compensate for data scarcity in low-resource environments.
- End-to-end systems: While this paper used a cascaded approach, exploring end-to-end ST models might be a future direction.
Summary updated