TL;DR
This paper presents the NICT-2 system for low-resource machine translation, which leverages a pre-trained multilingual mBART model. By expanding mBART to include previously unsupported languages and fine-tuning it on specific task data, the system achieved significant improvements in translation quality, demonstrating the effectiveness of pre-trained models for low-resource scenarios.
What Problem Does This Paper Address?
Machine translation (MT) systems often struggle with low-resource language pairs, where parallel corpora (pairs of sentences in two languages) are scarce. This scarcity limits the ability of standard MT models to learn effective translations. The NICT-SAP shared task specifically focuses on such low-resource language pairs, presenting a challenge for developing high-quality translation systems. Existing pre-trained models, while powerful, may not cover all the languages required for a specific task, necessitating further adaptation.
Methods
The core of the NICT-2 system is the mBART (multilingual BART) model, a pre-trained encoder-decoder Transformer architecture. The authors started with the publicly available mBART model, which supports 25 languages. A key innovation was expanding mBART's capabilities by adding support for Indonesian, Malay, and Thai, which were not included in the original model. This expansion was achieved through additional pre-training on monolingual corpora extracted from Wikipedia for these languages.
After this expansion, the model was fine-tuned using the parallel corpora provided for the NICT-SAP task. The paper also explored two additional techniques:
- Multilingual Models: Training a single model to handle all language pairs by using language tags.
- Domain Adaptation: Investigating two methods:
- Multi-domain models: Training a single model with domain tags.
- Fine-tuning-based adaptation: Training a mixture model and then fine-tuning separate domain-specific models.
Datasets
The primary datasets used were:
- NICT-SAP Task Datasets: These comprised parallel corpora for English to Hindi (Hi), Indonesian (Id), Malay (Ms), and Thai (Th). The task included two domains:
- ALT (Asian Language Translation) domain: Derived from WikiNews.
- IT (Information Technology) domain: Consisting of software document translations, using sub-corpora from OPUS (GNOME, KDE, Ubuntu).
- Wikipedia Corpora: Used for the additional pre-training of the mBART model for the unsupported languages (Indonesian, Malay, Thai).
- Common Crawl Corpus: The original source for the pre-trained mBART model.
Key Insights and Ideas
- Pre-trained models are crucial for low-resource MT: The paper strongly demonstrates that leveraging large, pre-trained multilingual models like mBART significantly boosts performance, even when fine-tuned on limited parallel data.
- Language expansion is vital: The ability to extend pre-trained models to cover new, low-resource languages is a critical step towards more inclusive MT systems. The additional pre-training on Wikipedia proved effective for this.
- Multilingual models' effectiveness is context-dependent: While multilingual models can be beneficial for low-resource scenarios by increasing training data size, their advantage diminishes when applied to already well-trained models like mBART. In such cases, they can even lead to performance degradation.
- Domain adaptation's impact is reduced by pre-training: The benefits of domain adaptation techniques (multi-domain models and fine-tuning) were less pronounced when the mBART model was already pre-trained. This suggests that the pre-trained model has already learned a broad understanding of language that generalizes well across domains.
Key Results
The NICT-2 system achieved significant improvements in BLEU scores compared to baseline systems that did not use the pretrained mBART model.
- For Indonesian, Malay, and Thai (languages not originally supported by mBART), the BLEU scores in the ALT domain increased by over 8 points.
- The paper reports that the expanded mBART model improved BLEU scores regardless of whether multilingual models or domain adaptation methods were applied.
- Specific BLEU scores are detailed in Tables 5 and 6, showing substantial gains for the submitted system (marked as "Submitted") over the "Baseline" for various language pairs and domains. For instance, in the ALT domain, English to Indonesian translation improved from a baseline BLEU of 15.25 to 33.31 with the full system.
Limitations and Future Work
- Limitations:
- The paper notes that the effectiveness of multilingual models was limited when used with the pretrained mBART model.
- Domain adaptation methods showed less impact when the pretrained model was used.
- Future Work: The paper does not explicitly detail future work, but the success of language expansion suggests that further efforts in this direction for even more languages would be beneficial. The findings also open avenues for exploring more efficient ways to combine pre-training with multilingual and domain adaptation strategies.
Summary updated