Dynamic Context-guided Capsule Network for Multimodal Machine Translation

Huan Lin, Fandong Meng, Jinsong Su, Yongjing Yin, Zhengyuan Yang, Yubin Ge, Jie Zhou, Jiebo LuoACM Multimedia 2020 · DOI · OpenReview

Abstract: Multimodal machine translation (MMT), which mainly focuses on enhancing text-only translation with visual features, has attracted considerable attention from both computer vision and natural language processing communities. Most current MMT models resort to attention mechanism, global context modeling or multimodal joint representation learning to utilize visual features. However, the attention mechanism lacks sufficient semantic interactions between modalities while the other two provide fixed visual context, which is unsuitable for modeling the observed variability when generating translation. To address the above issues, in this paper, we propose a novel Dynamic Context-guided Capsule Network (DCCN) for MMT. Specifically, at each timestep of decoding, we first employ the conventional source-target attention to produce a timestep-specific source-side context vector. Next, DCCN takes this vector as input and uses it to guide the iterative extraction of related visual features via a context-guided dynamic routing mechanism. Particularly, we represent the input image with global and regional visual features, we introduce two parallel DCCNs to model multimodal context vectors with visual features at different granularities. Finally, we obtain two multimodal context vectors, which are fused and incorporated into the decoder for the prediction of the target word. Experimental results on the Multi30K dataset of English-to-German and English-to-French translation demonstrate the superiority of DCCN.

Introduction

Multimodal Machine Translation (MMT) aims to improve translation accuracy by leveraging visual information alongside textual input. This is particularly critical for resolving linguistic ambiguities that text alone cannot clarify. For instance, the English word "bank" can refer to a financial institution or a sloping riverbank. Without visual context, a translation model might choose the wrong target term.

Figure 2: An example of multimodal machine translation where visual context helps disambiguate the word "bank" as a dusty riverbank (Erdwall).

Existing MMT approaches often suffer from three major flaws: insufficient semantic interaction between modalities, static visual context that does not evolve with the translation sequence, and a tendency to overfit due to the high parameter count of complex attention layers.

The DCCN Architecture

To address these limitations, we propose the Dynamic Context-guided Capsule Network (DCCN). Integrated into a standard Transformer architecture, DCCN dynamically "queries" the image during the decoding process to extract relevant visual information for each specific word being generated.

Figure 5: The overall architecture of the proposed DCCN model, showing the integration of global and regional visual features into the Transformer decoder.

Context-Guided Routing

Unlike traditional Capsule Networks that rely on spatial agreement to cluster features, DCCN uses the timestep-specific source-side context vector (derived from the text) as a guiding signal. This signal directs an iterative "routing" process to identify which visual features (capsules) are most relevant to the current translation step. This allows the text to act as a filter, suppressing irrelevant visual noise.

Dual-Granularity Visual Features

The model employs two parallel DCCNs to process visual data at different levels of abstraction:

  • Global Features: Extracted via ResNet-50, these represent high-level scene concepts and are particularly effective for translating verbs and adjectives.
  • Regional Features: Extracted via an R-CNN (bottom-up attention), these focus on specific objects (e.g., "dog," "ball") and are vital for disambiguating nouns.

A learnable gating mechanism then fuses the resulting multimodal context vectors, allowing the decoder to prioritize the most relevant granularity for the current word prediction.

Experimental Results

We evaluated DCCN on the Multi30K dataset for English-to-German (EN-DE) and English-to-French (EN-FR) tasks. The model consistently outperformed text-only Transformers and established MMT baselines while adding only approximately 1.0M extra parameters.

  • EN-DE Task: On the Test2016 set, DCCN achieved a 39.7 BLEU score, a significant improvement over the text-only Transformer (38.4) and the Doubly-attention model (38.7).
  • EN-FR Task: DCCN reached 61.2 BLEU on Test2016, outperforming the baseline by 1.1 points.
  • Robustness: The model also demonstrated superior performance on the WMT2017 and MSCOCO out-of-domain test sets, proving its ability to handle complex and ambiguous linguistic scenarios.

Conclusion

The Dynamic Context-guided Capsule Network represents a significant step forward in MMT by treating text as a dynamic query for visual information. By utilizing iterative capsule routing instead of standard attention, DCCN effectively filters noise and captures deep semantic interactions. Future work will explore applying this context-guided mechanism to other multimodal tasks such as Visual Question Answering (VQA) and multimodal summarization.

Summary updated

Abstract

Multimodal machine translation (MMT), which mainly focuses on enhancing text-only translation with visual features, has attracted considerable attention from both computer vision and natural language processing communities. Most current MMT models resort to attention mechanism, global context modeling or multimodal joint representation learning to utilize visual features. However, the attention mechanism lacks sufficient semantic interactions between modalities while the other two provide fixed visual context, which is unsuitable for modeling the observed variability when generating translation. To address the above issues, in this paper, we propose a novel Dynamic Context-guided Capsule Network (DCCN) for MMT. Specifically, at each timestep of decoding, we first employ the conventional source-target attention to produce a timestep-specific source-side context vector. Next, DCCN takes this vector as input and uses it to guide the iterative extraction of related visual features via a context-guided dynamic routing mechanism. Particularly, we represent the input image with global and regional visual features, we introduce two parallel DCCNs to model multimodal context vectors with visual features at different granularities. Finally, we obtain two multimodal context vectors, which are fused and incorporated into the decoder for the prediction of the target word. Experimental results on the Multi30K dataset of English-to-German and English-to-French translation demonstrate the superiority of DCCN. Our code is available on https://github.com/DeepLearnXMU/MM-DCCN.
BibTeX
@inproceedings{lin2020dynamic,
  author = {Huan Lin and Fandong Meng and Jinsong Su and Yongjing Yin and Zhengyuan Yang and Yubin Ge and Jie Zhou and Jiebo Luo},
  title = {{Dynamic Context-guided Capsule Network for Multimodal Machine Translation}},
  booktitle = {ACM Multimedia},
  year = {2020},
  doi = {10.1145/3394171.3413715},
}
Figure