Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)

Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Alan Yuille

Introduction

Historically, image captioning systems relied heavily on retrieval-based methods. Given a new image, these systems would search a database for the most visually similar image and simply return its existing caption. While effective for common, well-represented scenes, these systems were inherently limited by their training data. They could not describe novel combinations of objects—such as "a dog riding a surfboard"—if that exact sentence did not already exist in the database.

Example of a complex visual scene that generative models aim to describe.

The Multimodal Recurrent Neural Network (m-RNN) model bridges the gap between computer vision and natural language processing to move beyond retrieval and toward generative captioning. By fusing features from a Deep Convolutional Neural Network (CNN) and a Recurrent Neural Network (RNN), the m-RNN can generate entirely novel, grammatically correct image descriptions word-by-word, achieving state-of-the-art performance on both caption generation and image-sentence retrieval tasks.

Architecture and Methods

The challenge of generative captioning lies in effectively fusing visual information with linguistic context so the model knows which word to generate next based on both the words it has already spoken and the content of the image. The m-RNN architecture addresses this using three interacting functional parts:

  1. The Language Model Part: This component uses an RNN to maintain the "temporal context" of the sentence. A key novelty here is the use of a two-layer word embedding system, which the authors found to be more efficient at learning dense word representations than the standard single-layer approach.
Illustration of the unfolded Recurrent Neural Network architecture.
  1. The Vision Part: A deep CNN—specifically architectures like AlexNet or VggNet—is used to extract a high-level feature vector from the target image.
  2. The Multimodal Layer: This is the core innovation of the m-RNN. Instead of merely using the image to initialize the RNN (a common strategy in early models), the m-RNN inputs the image feature vector into this multimodal layer at every time step. This layer combines the current word's embedding, the RNN's hidden state, and the image features to predict the next word in the sequence. By feeding the image features at every step, the model doesn't "forget" the image content over long sentences, allowing the RNN's capacity to be dedicated entirely to linguistic context.

The model is trained using a log-likelihood cost function (minimizing perplexity) and utilizes the Rectified Linear Unit (ReLU) activation function to mitigate the vanishing gradient problems common in traditional RNNs.

Key Insights and Results

The m-RNN model was validated on four major benchmark datasets: IAPR TC-12, Flickr 8K, Flickr 30K, and MS COCO (the largest at the time, containing over 120,000 images). It set new state-of-the-art records across all four datasets at the time of publication.

  • Generation Performance: On the IAPR TC-12 dataset, the model achieved a BLEU-4 score of 0.208, significantly outperforming previous methods like the Log-BiLinear model.
  • Retrieval Performance: In image-to-sentence retrieval on MS COCO, the model achieved a Recall@1 of 41.0%, nearly doubling the performance of many contemporary baselines.
  • True Generative Novelty: Proving the model learned to "speak" rather than just memorize, the authors noted that 71% of the sentences generated for the MS COCO dataset were entirely novel and not present in the training set.
  • Consensus Reranking: The authors also introduced a "nearest neighbor" strategy. By examining the captions of images visually similar to the target, they could "rerank" the generated hypotheses to pick the one that best matched the consensus of the neighborhood, leading to a massive boost in BLEU scores.

Notably, the same m-RNN architecture functions effectively as a multitask framework, capable of generating new captions, finding the best image for a given sentence, and finding the best sentence for a given image.

Limitations and Future Work

While groundbreaking, the authors noted several areas for future improvement:

  • Fixed Vision Features: In this study, the CNN was pre-trained and its weights were "frozen" during the m-RNN training due to data constraints. Fine-tuning the CNN alongside the RNN on larger datasets would likely yield even better representations.
  • Simple RNN vs. LSTM: The authors used a simple RNN with ReLUs for efficiency and a smaller parameter count. They acknowledge that more complex units like Long Short-Term Memory (LSTM) networks might better handle extremely long-range linguistic dependencies.
  • Object-Level Features: The model primarily relies on global image features. Future iterations could incorporate object detection results (e.g., via R-CNN) to provide the model with more granular, localized information about specific items interacting in the scene.

Summary updated

Abstract

In this paper, we present a multimodal Recurrent Neural Network (m-RNN) model for generating novel image captions. It directly models the probability distribution of generating a word given previous words and an image. Image captions are generated by sampling from this distribution. The model consists of two sub-networks: a deep recurrent neural network for sentences and a deep convolutional network for images. These two sub-networks interact with each other in a multimodal layer to form the whole m-RNN model. The effectiveness of our model is validated on four benchmark datasets (IAPR TC-12, Flickr 8K, Flickr 30K and MS COCO). Our model outperforms the state-of-the-art methods. In addition, we apply the m-RNN model to retrieval tasks for retrieving images or sentences, and achieves significant performance improvement over the state-of-the-art methods which directly optimize the ranking objective function for retrieval. The project page of this work is: www.stat.ucla.edu/~junhua.mao/m-RNN.html .
BibTeX
@inproceedings{mao2015deep,
  author = {Junhua Mao and Wei Xu and Yi Yang and Jiang Wang and Alan Yuille},
  title = {{Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)}},
  booktitle = {International Conference on Learning Representations},
  year = {2015},
  url = {http://arxiv.org/abs/1412.6632},
}
Figure