TL;DR
This paper details the Idiap research institute's winning submission to the WAT 2019 English-Hindi multimodal translation task. Their text-only Transformer model, augmented with a large parallel corpus, outperformed all competitors, including those that incorporated visual information, highlighting the critical role of data quantity in low-resource translation.
What Problem Does This Paper Address?
The paper tackles the challenge of machine translation for low-resource language pairs, specifically English to Hindi. While multimodal translation (using images alongside text) holds promise for disambiguating words and improving translation quality, its actual benefit in machine translation is still debated. The authors aim to demonstrate the effectiveness of a strong text-based translation system, even when competing against multimodal approaches, by participating in the WAT 2019 task. They investigate whether abundant parallel text data can compensate for or even surpass the gains from visual context in this specific scenario.
Methods
The Idiap system is built upon the state-of-the-art Transformer model, a widely adopted neural network architecture for sequence-to-sequence tasks like translation. They utilized the OpenNMT-py implementation of the Transformer. For preprocessing, they employed the word pieces algorithm for subword tokenization, creating a shared vocabulary of 32,000 subword units for both English and Hindi.
The training strategy involved using a single GPU and a standard "Noam" learning rate schedule with warm-up steps. Crucially, they concatenated the official training data (HVG) with the large IITB English-Hindi parallel corpus and shuffled it at the sentence level. This augmentation with a significantly larger text corpus is a key aspect of their approach. They trained the model for up to 200,000 steps.
Datasets
The primary datasets used are:
- Hindi Visual Genome (HVG): This is the official training dataset provided by the WAT 2019 task organizers. It consists of English and Hindi short captions for image regions. It also includes development (D-Test), evaluation (E-Test), and challenge test sets (C-Test). The authors did not use the image features from this dataset.
- IITB English-Hindi Parallel Corpus: This is a large, publicly available parallel corpus containing approximately 1.4 million parallel segments. The authors found this corpus to be highly effective for English-Hindi translation and used it to augment the HVG training data.
Key Insights and Ideas
- Data is King (Even for Multimodal Tasks): The most striking insight is that their text-only system, heavily boosted by the IITB corpus, outperformed multimodal systems. This suggests that for certain language pairs and tasks, the sheer volume and quality of parallel text data can be more impactful than incorporating visual information, especially if the multimodal datasets are smaller or less diverse.
- BLEU Score Discrepancies: The authors highlight a significant difference between BLEU scores calculated by the Moses toolkit and sacreBLEU. This discrepancy, attributed to tokenization differences, underscores the importance of consistent evaluation metrics and reporting practices, especially for less-resourced languages. They found the Moses implementation yielded scores ~10 points higher, which they cautiously interpret as potentially indicating better robustness.
- Text-Only vs. Multimodal Evaluation: The paper demonstrates that even when the underlying reference translations are identical, a text-only system can achieve superior results compared to multimodal systems in automatic evaluations. This raises questions about the current effectiveness and integration of visual features in multimodal MT.
Key Results
The Idiap system achieved the top rank in the text-only track of the WAT 2019 English-Hindi Multimodal Translation Task, both in automatic and manual evaluations.
- E-Test (Evaluation Set): Their text-only system achieved a BLEU score of 41.32, which was higher than the best competitor in the multimodal track (40.55).
- C-Test (Challenge Set): Their text-only system achieved a BLEU score of 30.94, again outperforming the best multimodal competitor (20.37).
- Outperforming Multimodal Systems: Notably, their text-only submission surpassed systems that did utilize image features in the multimodal track.
Limitations and Future Work
- Manual Evaluation Comparability: The authors acknowledge that comparing manual evaluation scores between the text-only and multimodal tracks is difficult because they were conducted in separate batches, potentially introducing bias.
- Sentence Length Mismatch: For the C-Test, they observed that the average sentence length was slightly longer than the training data. This led to a brevity penalty affecting BLEU scores, suggesting that NMT models can struggle to produce outputs longer than what they were trained on.
- Future Work: The team plans to:
- Integrate image features into their system and conduct a comparative study with their current text-only setup.
- Experiment with the image captioning variant of the task.
- Investigate the impact of using visual features on the translation of particularly ambiguous words.
Summary updated