TL;DR
This paper explores how to improve Neural Machine Translation (NMT) for similar languages with limited parallel data. The authors propose three NMT systems for Hindi-Nepali, including a semi-supervised approach leveraging monolingual data and a novel method of augmenting parallel data by copying sentences. Surprisingly, the baseline system using only parallel data outperformed the more complex methods.
What Problem Does This Paper Address?
Machine translation systems, especially Neural Machine Translation (NMT), typically require vast amounts of parallel text (sentences and their translations) to achieve high accuracy. However, for many language pairs, particularly those that are similar but less widely spoken, this parallel data is scarce. This scarcity hinders the development of effective NMT systems. The WMT 2019 "Similar Language Translation Task" specifically aims to address this by encouraging the use of monolingual data (text in a single language) to boost translation quality when parallel data is limited. This paper tackles this challenge by investigating different strategies for utilizing monolingual data for the Hindi-Nepali language pair, which are Indo-Aryan languages and share significant similarities.
Methods
The authors submitted three systems for the Hindi-Nepali translation task, all based on a Recursive Neural Network (RNN) encoder-decoder architecture.
- Pure RNN-based NMT system: This is the baseline system, trained solely on the available parallel Hindi-Nepali corpus.
- Semi-supervised RNN-based NMT system: This system utilizes monolingual data from both Hindi and Nepali. It employs an architecture similar to that proposed by Artetxe et al. (2017), which features a shared encoder and separate decoders for each language. The training alternates between denoising (reconstructing corrupted sentences) and back-translation (translating from the target language back to the source language) to leverage monolingual data. This architecture can also be used for completely unsupervised NMT.
- Utilization of copied data in RNN-based NMT: This system augments the existing parallel data by creating synthetic data. Specifically, it copies source sentences to the target side and target sentences to the source side. This effectively triples the amount of training data without using any external monolingual corpora. The original and copied data are then combined and shuffled for training.
All systems were implemented using the framework from Artetxe et al. (2017).
Datasets
- Hindi-Nepali Parallel Data: Provided by the task, containing 65,505 sentences.
- IITB Hindi Monolingual Corpora: A large collection of Hindi text (45,075,242 sentences).
- Nepali Monolingual Corpora: Created from Wikipedia and Common-Crawl data by filtering sentences and keeping those less than 500 words long (6,688,559 sentences).
Key Insights and Ideas
- Surprising Baseline Performance: The most striking finding is that the simple RNN-based NMT system trained only on parallel data outperformed both the semi-supervised system using monolingual data and the system augmented with copied data. This suggests that for similar languages like Hindi-Nepali, even a small amount of high-quality parallel data might be more effective than complex methods attempting to leverage monolingual resources, at least with the specific configurations used.
- Confusion from Copied Data: The system that augmented parallel data by copying sentences performed poorly. The authors hypothesize that this is because the model became confused by receiving both original and copied sentences without any distinction, potentially leading to errors in learning the correct translation mappings.
- Potential of Semi-Supervised NMT: While the semi-supervised system didn't outperform the baseline, it represents a standard approach for leveraging monolingual data. The paper highlights the need for further study into the size and quality of monolingual data to understand its impact.
Key Results
The paper reports BLEU scores for the test and development sets.
- Baseline (Parallel Data Only): Test BLEU: 3.5, Dev BLEU: 4.6
- Semi-supervised (With Monolingual Data): Test BLEU: 2.8, Dev BLEU: 3.27
- Augmented (With Copy Data): Test BLEU: 2.7, Dev BLEU: 4.38
As evident, the "Basic" system (parallel data only) achieved the highest BLEU score on the test set, while the "With copy data" system had a surprisingly high dev score but a low test score. The "With Monolingual Data" system performed the lowest on both metrics.
Limitations and Future Work
- Data Confusion: The authors acknowledge that the poor performance of the copied data system might be due to the lack of distinction between original and copied sentences during training, leading to confusion.
- Monolingual Data Analysis: A deeper study is needed to understand the impact of the size and quality of monolingual data on the performance of semi-supervised NMT systems.
- Future Work: The authors suggest that future work should focus on analyzing the size and quality of monolingual data to better understand its role in improving translation performance. They also imply that better methods for distinguishing and utilizing augmented data might be necessary.
Summary updated