TL;DR
This paper tackles the challenge of low-resource language translation by enhancing English-to-Hindi multimodal neural machine translation (NMT). They achieve this by incorporating phrase pair injection, a data augmentation technique, into their multimodal NMT system, leading to improved translation quality and a second-place finish in the WAT2021 challenge.
What Problem Does This Paper Address?
Machine translation (MT) has seen significant advancements with Neural Machine Translation (NMT), but its effectiveness is heavily reliant on large amounts of parallel training data. This becomes a major hurdle for "low-resource" language pairs, where such data is scarce. To combat this, researchers have explored "multimodal NMT" (MNMT), which leverages additional modalities like images alongside text to enrich the translation process. This paper focuses on the English-to-Hindi language pair, a classic low-resource scenario, and aims to improve MNMT performance by addressing the data scarcity issue.
Methods
The core of this work lies in enhancing the training data for their NMT models. They employ a phrase pair injection strategy, inspired by previous work, to augment the existing parallel corpus. This involves extracting phrase pairs from a source-side monolingual corpus using SMT techniques (specifically, Giza++ for word alignment) and then incorporating these into the training data.
For the multimodal aspect, they utilize a pre-trained Convolutional Neural Network (CNN) with VGG19 to extract visual features from images. These visual features are then combined with textual features. For text representation, they use GloVe word embeddings.
The NMT architecture itself is based on the OpenNMT-py toolkit. They use a bidirectional RNN (BRNN) as the encoder and a doubly-attentive RNN as the decoder, following established settings. The multimodal NMT model is trained for 40 epochs, while the text-only model is trained for a much longer duration (100,000 steps). A key difference highlighted is the use of phrase pairs in the augmented parallel data for this work, compared to their previous efforts.
Datasets
The primary dataset used is the Hindi Visual Genome 1.1 dataset, provided by the WAT2021 organizers. This dataset includes:
- Training data: 28,927 parallel English-Hindi sentences and corresponding images.
- Test (Evaluation Set): 1,595 parallel sentences and images.
- Test (Challenge Set): 1,400 parallel sentences and images.
- Validation data: 998 parallel sentences and images.
Additionally, they utilize:
- Monolingual data: A substantial amount of English (107,597,494 sentences) and Hindi (44,949,045 sentences) monolingual data from the IIT Bombay English-Hindi parallel corpus and WMT16.
Key Insights and Ideas
- Phrase Pair Injection as a Data Augmentation Lever: The paper strongly suggests that augmenting parallel data with extracted phrase pairs from monolingual sources can significantly boost translation performance, especially in low-resource settings. This is a practical way to "synthetically" increase the training data's richness.
- Multimodality's Advantage in Low-Resource Scenarios: The results consistently show that the multimodal NMT model outperforms its text-only counterpart. This reinforces the idea that leveraging visual information can compensate for the lack of extensive textual parallel data in low-resource language pairs.
- Incremental Improvement is Key: The authors demonstrate that their current approach, building upon their previous work, yields substantial improvements. This highlights the iterative nature of research in this field, where refining techniques like data augmentation can lead to significant gains.
Key Results
The paper reports strong performance on the WAT2021 English to Hindi multimodal translation task:
- Second Place Finish: Their multimodal submission achieved the second rank in the challenge test set.
- Quantitative Metrics:
- Multimodal NMT (Challenge Set): BLEU score of 39.28, RIBES of 0.792097, and AMFM of 0.830230.
- Text-only NMT (Challenge Set): BLEU score of 37.16, RIBES of 0.770621, and AMFM of 0.798670.
- Comparison to Previous Work: Their current results show significant improvements over their WAT2020 submission, with notable increases in BLEU, RIBES, and AMFM scores for both multimodal and text-only systems.
Limitations and Future Work
- Limitations: The paper doesn't explicitly detail limitations of their current approach, but the reliance on pre-trained CNNs and GloVe embeddings implies a dependency on the quality of these pre-trained models. The phrase extraction process, while effective, might also introduce noise or suboptimal phrase pairs.
- Future Work: The authors explicitly state their intention to "investigate a multilingual approach to improve the performance of multimodal NMT" in future work. This suggests exploring how knowledge from multiple languages can be leveraged to further enhance multimodal translation capabilities.
Summary updated