Feature-level Incongruence Reduction for Multimodal Translation

Zhifeng Li, Yu Hong, Yuchen Pan, Jian Tang, Jianmin Yao, Guodong Zhou2021 · DOI

Abstract

Caption translation aims to translate image annotations (captions for short). Recently, Multimodal Neural Machine Translation (MNMT) has been explored as the essential solution. Besides linguistic features in captions, MNMT allows visual (image) features to be used. The integration of multimodal features reinforces the semantic representation and considerably improves translation performance. However, MNMT suffers from the incongruence between visual and linguistic features. To overcome the problem, we propose to extend MNMT architecture with a harmonization network, which harmonizes multimodal features (linguistic and visual features) by unidirectional modal space conversion. It enables multimodal translation to be carried out in a seemingly monomodal translation pipeline. We experiment on the golden Multi30k-16 and 17. Experimental results show that, compared to the baseline, the proposed method yields improvements of 2.2% BLEU for English-to-German (En-De), 7.6% for English-to-French (En-Fr), and 1.5% for English-to-Czech (En-Cz). The utilization of a harmonization network leads to performance competitive with the state-of-the-art.

Addressing the Modality Barrier

Multimodal Neural Machine Translation (MNMT) aims to improve translation accuracy by providing the model with both a source sentence and a related image. However, a fundamental "incongruence" exists between these two data types. Linguistic features are sequential and dependent on syntax and grammar, while visual features are spatial and sequence-independent.

When a model attempts to combine raw image vectors with text embeddings, they often fail to correspond because they exist in incompatible semantic spaces. This "modality barrier" prevents the translation model from fully utilizing visual context to disambiguate words or refine meaning. To solve this, this research introduces a Harmonization Network designed to bridge the gap between vision and language.

Methods: The Detour Strategy

The core innovation is a "detour" strategy. Instead of feeding raw visual features directly into the translation network, the researchers use a Harmonization Network based on an image captioning (CAP) model.

The "detour" strategy uses a harmonization network (CAP) to convert visual features into linguistic features before they reach the translation engine.
  1. Feature Transformation: The system employs a top-down attentive captioning model. Rather than using the final text output, it extracts the hidden states produced during the captioning process. These states represent "image-dependent attention-aware linguistic features"—essentially, the image is pre-processed into a format that "looks" like language to the translation model.
  2. Integration Variants: The study tested several ways to integrate these harmonized features into a standard Attentive Encoder-Decoder NMT baseline, including blending features with the encoder's hidden states (CAP-ATT), initializing the encoder (CAP-ENC) or decoder (CAP-DEC), and modulating target-language word embeddings (CAP-TKN).

Experimental Results

The researchers evaluated their approach on the Multi30k (2016 & 2017) benchmarks, using MSCOCO for pre-training the CAP model and ResNet-101 for the image encoder.

Experimental results showing BLEU, METEOR, and TER scores for English-to-German (En→De) and English-to-French (En→Fr) translation across different integration strategies.

The proposed method consistently outperformed the monomodal baseline:

  • English-to-French (En→Fr): Achieved a significant 7.6% BLEU improvement over the baseline on Multi30k-16.
  • English-to-German (En→De): Saw a 2.2% BLEU improvement.
  • Visual Disambiguation: On the Ambiguous COCO dataset, the model successfully used images to choose correct verbs when text alone was ambiguous (e.g., determining if "pulling" refers to horses or dogs).

Key Insights and Conclusion

By converting vision into a linguistic-like representation before fusion, the NMT model avoids the difficulty of learning to interpret raw pixels and syntax simultaneously. Furthermore, because the CAP model produces a sequence of hidden states, it aligns naturally with the sequential nature of source text.

While highly effective, the authors noted that performance varies by target language, suggesting that the "strictness" of a language's syntax (like French) might make it more compatible with these feature vectors than morphologically complex languages. Future work will investigate these syntactic influences and explore Generative Adversarial Network (GAN) based models to further refine feature adaptation.

Summary updated

Abstract

Caption translation aims to translate image annotations (captions for short). Recently, Multimodal Neural Machine Translation (MNMT) has been explored as the essential solution. Besides of linguistic features in captions, MNMT allows visual (image) featuresto be used. The integration of multimodal features reinforces the semantic representation and considerably improves translation performance. However, MNMT suffers from the incongruence between visual and linguistic features. To overcome the problem, we propose to extend MNMT architecture with a harmonization network, which harmonizes multimodal features (linguistic and visual features) by unidirectional modal space conversion. It enables multimodal translation to be carried out in a seemingly monomodal translation pipeline. We experiment on the golden Multi30k-16 and 17. Experimental results show that, compared to the baseline, the proposed method yields the improvements of 2.2% BLEU for the scenario of translating English captions into German (EnDe) at best, 7.6% for the case of Englishto-French translation (EnFr) and 1.5% for English-to-Czech (EnCz). The utilization of harmonization network leads to the competitive performance to the-state-of-the-art.
BibTeX
@misc{li2021feature,
  author = {Zhifeng Li and Yu Hong and Yuchen Pan and Jian Tang and Jianmin Yao and Guodong Zhou},
  title = {{Feature-level Incongruence Reduction for Multimodal Translation}},
  year = {2021},
  doi = {10.18653/v1/2021.alvr-1.1},
}
Figure