TL;DR
This paper presents the CUNI system for the WMT18 Multimodal Translation Task, which aims to improve machine translation by incorporating visual information. They propose two novel methods using the Transformer architecture: one that directly integrates image features into the decoder and another that uses an "imagination" component to predict visual features as an auxiliary objective. Both methods significantly improve translation quality over strong textual baselines, especially when combined with extensive data augmentation.
What Problem Does This Paper Address?
Traditional machine translation systems primarily rely on textual input. However, language often refers to real-world concepts that are visually grounded. Multimodal Machine Translation (MMT) seeks to leverage this visual grounding to disambiguate meaning and improve translation accuracy. The WMT Multimodal Translation Task specifically focuses on generating image descriptions (captions) in a target language, given a source language caption and the image itself. While previous work has explored using recurrent neural networks (RNNs) for MMT, the authors note that self-attention-based models like the Transformer have become state-of-the-art in standard neural machine translation. The challenge is to effectively integrate visual information into these powerful Transformer architectures to achieve better translation performance, a task that has proven difficult for RNN-based approaches.
Methods
The CUNI system builds upon the Transformer architecture, a popular choice for sequence-to-sequence tasks. They propose two main ways to incorporate visual information:
- Doubly Attentive Transformer: This method modifies the Transformer decoder by adding an extra attention sub-layer. This new sub-layer allows the decoder to attend to image features directly, in addition to attending to the encoded textual information. This creates a "doubly attentive" decoder that can jointly consider both modalities.
- Imagination Component: Inspired by previous work on RNNs, this approach uses an auxiliary training objective. The model is trained to predict visual features of the image from the textual encoder's representation. This "imagination" component acts as a regularizer for the encoder, encouraging it to learn representations that are semantically aligned with the visual content.
Both methods are trained using a combination of translation and, in the case of the imagination component, the auxiliary visual prediction loss.
Datasets
The primary dataset used is Multi30k, which is an extension of Flickr30k. It contains approximately 29,000 training images, 1,014 validation images, and 1,000 test images, each with six crowd-sourced captions in multiple languages.
To address the relatively small size of Multi30k, the authors also acquired additional data:
- Synthetic data for German: Generated using back-translation from the WMT16 MMT Task 2 training dataset.
- Pseudo in-domain data for Czech and German: Filtered from general domain corpora.
- Synthetic parallel data for Czech and German: Created using back-translation from large monolingual corpora (Common Crawl, News Crawl).
- EU Bookshop: An out-of-domain parallel dataset.
For training the imagination component, the MSCOCO dataset was used, which contains 414k images with English captions.
Key Insights and Ideas
- Transformer's Advantage: The paper demonstrates that the Transformer architecture, when adapted for MMT, can outperform RNN-based models.
- Visual Information is Crucial: Explicitly incorporating visual information, whether directly in the decoder or as an auxiliary objective, leads to significant improvements in translation quality. This is a key finding, as previous RNN-based multimodal systems showed only minor gains.
- Data Augmentation is Key: The most substantial gains were achieved by training on the augmented datasets, highlighting the importance of diverse and large-scale data for multimodal tasks.
- Imagination as a Regularizer: The "imagination" component effectively regularizes the encoder, forcing it to learn richer representations that capture visual semantics, leading to better translation.
- Adversarial Evaluation: An adversarial evaluation using "fake" images confirmed that the multimodal models indeed utilize visual information during inference, though the effect varied across languages.
Key Results
- The doubly-attentive decoder performed best in the constrained setup (using only Multi30k data).
- Training on additional parallel data (unconstrained setup) yielded the biggest performance gains.
- The imagination architecture outperformed purely textual models.
- Quantitative results (BLEU scores) showed significant improvements over textual baselines. For example, in the en-cs translation task, the imagination model achieved a BLEU score of 36.3/32.8 (single/averaged) compared to a textual baseline of 31.2/30.1.
- Qualitative analysis revealed that the models, especially for Czech, sometimes struggled with morphology, but this was mitigated by using lemmatized outputs.
Limitations and Future Work
- Morphological Issues: The models, particularly for Czech, showed some limitations in capturing morphology, although this was partially addressed by using lemmatized outputs.
- Language-Specific Data Acquisition: The authors hypothesize that the German models were less affected by the adversarial evaluation due to differences in training data acquisition compared to French and Czech. This suggests that data characteristics can influence the effectiveness of multimodal integration.
- Future Directions: The paper implicitly suggests that further exploration of data augmentation strategies and architectural modifications for better visual-textual alignment could lead to even greater improvements. The success of the imagination component also points towards exploring more sophisticated auxiliary tasks for regularizing the encoder.
Summary updated