The AFRL-Ohio State WMT18 Multimodal System: Combining Visual with Traditional

Jeremy Gwinnup, Joshua Sandvick, Michael Hutt, Grant Erdmann, John Duselis, James DavisEMNLP 2018

TL;DR

This paper presents a multimodal machine translation (MMT) system that combines visual information with traditional text-based translation. By integrating a visual machine translation (VMT) component into a system ensemble, they aim to improve translation quality, particularly for image-caption pairs, and explore the potential of visual input to rival text-only approaches.

What Problem Does This Paper Address?

Traditional machine translation (MT) systems primarily rely on textual input. However, many real-world scenarios involve multimodal information, such as images paired with text. The challenge lies in effectively leveraging this visual information to enhance translation accuracy and fluency. While previous work has explored using visual context to aid MT, this paper specifically investigates the viability of a Visual Machine Translation (VMT) system, where the image itself drives the translation process, and how its integration with established MT systems can lead to improved overall performance. They aim to see if visual input can achieve parity with traditional text-based MT.

Methods

The AFRL-Ohio State WMT18 submission builds upon their previous VMT system. The core of their approach involves two main components:

  • Visual Machine Translation (VMT) System: This system uses a "captionator" trained on image-caption datasets (like Flickr30k and ImageCLEF) to generate candidate target language sentences based on visual input. A learned mapping function encodes the source language caption and maps it to the most probable candidate captions in the target language.
  • Caption Selection Mechanism: A revised neural network, centered around a two-sided Long Short-Term Memory (LSTM) encoder, is used to match English and German sentences. This mechanism achieved high accuracy in correctly matching captions.
  • System Combination: To leverage the strengths of various MT approaches, they combined their VMT system with three traditional MT systems: OpenNMT, Marian, and Moses. This ensemble approach, using RWTH's Jane System combination, aims to create a more robust and consensual final translation by introducing variability from different MT outputs.

Datasets

  • Training:
    • Google Show and Tell captionator was trained on 29,000 training image+German caption pairs, augmented with 20,000 image+German captions from ImageCLEF 2008.
  • Additional models were trained on a constrained set of 29,000 WMT pairs, with single or five captions per image.
  • The caption selection mechanism was trained on WMT18 Multimodal Task 1 English and German training sentences from the 2018 Multi30k dataset.
  • Traditional MT systems (OpenNMT, Marian, Moses) were trained on approximately 41 million parallel lines of preprocessed German-English data provided by the WMT18 organizers.
  • Evaluation:
    • The caption selection mechanism was tested on the 2018 Multi30k datasets.
  • A posteriori analysis for captionator output scoring was performed on the WMT17 multimodal validation set.

Key Insights and Ideas

  • Visual Input as a Peer: The paper posits that visual domain can approach parity with traditional text-based MT systems, moving beyond just enhancement to potentially driving translation.
  • Ensemble Power: Combining a VMT system with multiple traditional MT systems (statistical and neural) shows a positive performance trend, suggesting that multimodal integration can boost overall translation quality.
  • Captionator Limitations: The "Show and Tell" captionator, while functional, produced a high number of "unknown word" tokens, leading to a bias towards shorter, generic captions. Augmenting with ImageCLEF data improved results.
  • Caption Selection Accuracy: The LSTM-based caption selection mechanism demonstrated high accuracy in matching English and German captions, indicating its effectiveness in aligning visual and textual representations.

Key Results

  • The 4Combo submission (VMT + Marian + Moses + OpenNMT) achieved a BLEU score of 24.3 and a METEOR score of 45.4, a significant improvement over their WMT17 submission.
  • The caption selection mechanism achieved 99.4% matching accuracy on the 29,000-image training set and 92.4% on the 1,014-image test set.
  • While the 4Combo system did not meet the baseline, the performance trend when adding the VMT system to traditional MT combinations was positive.

Limitations and Future Work

  • Limitations:
    • The VMT system's performance is inherently limited by the captionator's ability to accurately describe visual content.
  • The system did not achieve baseline performance in the WMT18 submission.
  • No post-editing was performed for any of the submission systems.
  • Future Work:
    • Employ a more generalized approach to estimate objects or concepts directly from images, rather than solely relying on sentence structure generation.
  • Utilize visual information to influence other MT systems more broadly, moving beyond just caption generation.
  • Explore using visual content to provide richer contextual information for MT.

Summary updated

BibTeX
@inproceedings{gwinnup2018afrl,
  author = {Jeremy Gwinnup and Joshua Sandvick and Michael Hutt and Grant Erdmann and John Duselis and James Davis},
  title = {{The AFRL-Ohio State WMT18 Multimodal System: Combining Visual with Traditional}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure