TL;DR
This paper presents multimodal neural machine translation (MMT) systems developed by LIUM and CVC for the WMT18 shared task. They introduce a "filtered attention" mechanism to better integrate visual features by dynamically masking them based on source sentence context, leading to improved translation quality, particularly for English-German pairs.
What Problem Does This Paper Address?
Multimodal Neural Machine Translation (MMT) aims to improve translation by leveraging information from both text and images. While previous approaches often integrated global visual features, the authors found that the qualitative analysis of visual attention was inconsistent and weak. They hypothesized this was due to diluted relevant visual information and competition with the more dominant source text. This paper addresses the challenge of effectively and parsimoniously integrating visual information into the translation process, specifically by refining the attention mechanism to focus on the most relevant visual regions.
Methods
The core contribution of this paper is the Filtered Attention (FA) mechanism. Building upon their previous multimodal attention (MA) architecture, FA introduces a novel way to filter convolutional visual features. This is achieved by:
- Encoder-guided Spatial Attention: A learnable masking operation is applied to the visual feature maps. This mask, denoted as $\beta_{pre}$, is computed using an attention mechanism that takes the source language encoder's last hidden state ($h_{enc}$) and the visual features ($V$) as input.
- Dynamic Masking: The computed mask $\beta_{pre}$ is then used to spatially filter the convolutional feature maps ($V$), creating a refined feature map ($V' = \beta_{pre} \odot V$). This filtered map is then used in the multimodal context vector computation.
- Feature Normalization: The authors also experimented with L2 normalization on the channel dimension of the convolutional feature maps. They found this crucial for preventing feature saturation in the early stages of training and significantly improved performance.
The overall architecture uses a standard encoder-decoder NMT framework with GRU layers. For multimodal systems, visual features are extracted using a pre-trained ResNet50 CNN.
Datasets
The primary dataset used is Multi30k, which contains English-French and English-German sentence pairs with corresponding images. The authors used the provided train, dev, test2016, and test2017 splits. They also created a new training split by concatenating the train and dev sets. For evaluation, they used the official test2017 and test2018 sets.
Key Insights and Ideas
- Visual Attention Needs Filtering: Simply integrating global visual features or even standard attention over them isn't enough. The visual information needs to be actively filtered to avoid noise and focus on relevant parts.
- Source Context is Key for Visual Filtering: Using the source sentence's representation to guide the filtering of visual features is a powerful idea. This allows the model to dynamically decide which parts of the image are most pertinent to the current translation step.
- Feature Normalization is Crucial: L2 normalization of convolutional features significantly boosts multimodal performance, suggesting that raw CNN features can be problematic for downstream NMT models without this preprocessing.
- MMT Doesn't Always Beat Monolingual NMT: While MMT aims to improve translation, the authors found that their multimodal attention system (MA) with L2 normalization only approached the performance of their baseline monomodal NMT system. The FA mechanism was necessary to achieve statistically significant improvements.
Key Results
- Filtered Attention (FA) Improves Performance: The FA mechanism consistently improved translation quality over the baseline multimodal attention (MA) system. For English-German, FA achieved a BLEU score of 31.6 ± 0.5 and METEOR of 52.5 ± 0.4 on test2017, which was statistically different from the baseline NMT.
- L2 Normalization is Essential: Without L2 normalization, the multimodal attention system performed worse than the baseline NMT. With normalization, performance improved by approximately 2 points in BLEU and METEOR.
- Image Size Impact is Marginal: While they experimented with different input image sizes (224x224 vs. 448x448), the impact was marginal, though they opted for the larger size for slight gains.
- English-French Performance Lag: The improvements seen in English-German were not replicated for English-French, possibly due to hyperparameter transfer without specific tuning for the French task.
- Ranking: Their final constrained submissions ranked first for English-French and second for English-German on the WMT18 task based on METEOR.
Limitations and Future Work
- English-French Performance: The authors acknowledge that the improvements for English-German did not translate to English-French, suggesting that hyperparameter tuning specific to the French task might be needed.
- Qualitative Analysis: While they mention qualitative analysis was initially unsatisfying, the paper focuses on quantitative results. A deeper qualitative analysis of why the FA mechanism works and where it focuses attention would be valuable.
- Human Evaluation: The paper notes that final submissions are primarily evaluated by human direct assessment, and their results are based on automatic metrics. Future work could involve detailed human evaluation to confirm the observed improvements.
- Future Work: The authors plan to qualitatively analyze the spatial attention and further improve it.
Summary updated