Image Captioning as Neural Machine Translation Task in SOCKEYE

Loris Bazzani, Tobias Domhan, Felix HieberarXiv 2018 · arXiv

TL;DR

Researchers from Amazon have integrated image captioning into SOCKEYE, an open-source Neural Machine Translation (NMT) toolkit. By treating image features as a "source language," they demonstrate that NMT architectures like Transformers and attentional RNNs are highly effective for generating descriptions, especially when paired with object detection features.

What Problem Does This Paper Address?

Image captioning sits at the intersection of computer vision (CV) and natural language processing (NLP). The core challenge is translating visual spatial information into a coherent, grammatically correct temporal sequence of words.

Historically, many models used the final fully-connected layer of a CNN as a global image descriptor. However, this approach discards crucial spatial information—knowing where an object is helps the model describe what it is doing. While attentional models have been proposed to solve this, the authors identify a need for a unified, scalable framework that can leverage the rapid advances in NMT (like the Transformer architecture) for the vision domain.

Methods

The authors treat image captioning as a sequence-to-sequence task, utilizing the SOCKEYE toolkit. Their approach consists of two main components:

  1. Image Encoder: They use pre-trained ResNets (ResNet-152 and ResNet-101). Instead of using a single global vector, they extract convolutional maps to preserve spatial information. They also experiment with "bottom-up" features—extracting mean-pooled features from specific object detections rather than a uniform grid.
  2. Caption Decoders: They benchmark three dominant NMT architectures:
  3. Attentional RNN (ARNN): An LSTM-based decoder using MLP-style attention.
  4. Self-Attentional Transformer (SAT): The now-standard architecture that replaces recurrence with multi-head self-attention.
  5. Fully-Convolutional Network (FCN): A decoder based entirely on convolutional layers.

The visual features are projected into a lower-dimensional space (512-d) to match the decoder's hidden state size before being processed.

Datasets

The models were trained and evaluated on two benchmark datasets:

  • MS-COCO: A large-scale dataset with over 120,000 images, each with 5 human-annotated captions.
  • Flickr30k: A smaller dataset containing approximately 31,000 images. The authors used the standard "Karpathy splits" for consistent evaluation against prior literature.

Key Insights and Ideas

  • NMT Parity: The paper confirms that image captioning can be effectively "solved" using standard NMT toolkits. The transition from RNNs to Transformers in NLP translates well to image-to-text tasks.
  • The Importance of Attention: Removing attention entirely (using only a global image vector) leads to a significant performance drop. Interestingly, MLP-style attention and Multi-head attention significantly outperformed simple dot-product attention, which the authors attribute to normalization issues between disparate visual and textual representations.
  • Objects over Pixels: The most substantial performance gains came not from changing the decoder architecture, but from changing the input features. Using features from an object detector (bottom-up) rather than raw convolutional grids (top-down) provided a major boost in CIDEr scores.
  • Small Tweaks Matter: Incremental improvements like adding weight normalization, concatenating global image descriptors to word embeddings, and optimizing beam size (reducing it from 5 to 3) collectively yielded better results.

Key Results

The study found that ARNN and SAT performed almost identically, both significantly outperforming the FCN decoder (see Table 1 in the paper).

On the MS-COCO dataset, their best model (ARNN with object detectors) achieved a CIDEr score of 1.098 and a B@4 (BLEU-4) of 0.339. While this was slightly lower than models specifically optimized using Reinforcement Learning (RL), it represented a >3% improvement over most standard cross-entropy trained models at the time.

Qualitative results (Figure 1) show the model's ability to identify complex interactions, such as a bird perched near a window or a tennis player in mid-swing.

A bird perched on a wire in front of a window.
Example of a successful caption prediction from the paper's qualitative analysis.

Limitations and Future Work

  • The Metric Gap: The models were trained using cross-entropy loss, which doesn't always correlate with human judgment or specialized metrics like CIDEr. The authors suggest that integrating Self-Critical Sequence Training (SCST)—a form of reinforcement learning—into SOCKEYE is a necessary next step to reach state-of-the-art performance.
  • Encoder Finetuning: In their experiments, finetuning the CNN encoder actually degraded performance. They note that this requires a very careful, multi-phase training schedule that they did not explore in this work.
  • Vocabulary Constraints: For smaller datasets like Flickr30k, the model remains sensitive to vocabulary frequency, requiring the removal of rare words to maintain stability.

Summary updated

Abstract

Image captioning is an interdisciplinary research problem that stands between computer vision and natural language processing. The task is to generate a textual description of the content of an image. The typical model used for image captioning is an encoder-decoder deep network, where the encoder captures the essence of an image while the decoder is responsible for generating a sentence describing the image. Attention mechanisms can be used to automatically focus the decoder on parts of the image which are relevant to predict the next word. In this paper, we explore different decoders and attentional models popular in neural machine translation, namely attentional recurrent neural networks, self-attentional transformers, and fully-convolutional networks, which represent the current state of the art of neural machine translation. The image captioning module is available as part of SOCKEYE at https://github.com/awslabs/sockeye which tutorial can be found at https://awslabs.github.io/sockeye/image_captioning.html .
BibTeX
@misc{bazzani2018image,
  author = {Loris Bazzani and Tobias Domhan and Felix Hieber},
  title = {{Image Captioning as Neural Machine Translation Task in SOCKEYE}},
  year = {2018},
  eprint = {1810.04101},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.04101},
}
Figure