Predicting Visual Features from Text for Image and Video Caption Retrieval

Jianfeng Dong, Xirong Li, Cees G. M. Snoek

Abstract

This paper strives to find amidst a set of sentences the one best describing the content of a given image or video. Different from existing works, which rely on a joint subspace for their image and video caption retrieval, we propose to do so in a visual space exclusively. Apart from this conceptual novelty, we contribute Word2VisualVec, a deep neural network architecture that learns to predict a visual feature representation from textual input. Example captions are encoded into a textual embedding based on multi-scale sentence vectorization and further transferred into a deep visual feature of choice via a simple multi-layer perceptron. We further generalize Word2VisualVec for video caption retrieval, by predicting from text both 3-D convolutional neural network features as well as a visual-audio representation. Experiments on Flickr8k, Flickr30k, the Microsoft Video Description dataset and the very recent NIST TrecVid challenge for video caption retrieval detail Word2VisualVec's properties, its benefit over textual embeddings, the potential for multimodal query composition and its state-of-the-art results.

Introduction: Bridging the Semantic Gap

The core challenge in cross-modal retrieval—finding the best caption for an image or vice versa—is the "semantic gap." Images are composed of pixels, while sentences are sequences of words. Historically, researchers bridged this gap by learning a joint latent subspace where both modalities could be compared.

Conceptual overview of mapping sentences into a visual feature space.

However, this paper argues that joint subspaces are unnecessarily complex. With the advent of powerful, pre-trained Convolutional Neural Networks (ConvNets) like ResNet and GoogLeNet, visual features have become highly discriminative and stable. The authors propose Word2VisualVec (W2VV), which teaches a text model to "imagine" what the corresponding visual feature vector should look like, mapping natural language directly into a high-dimensional visual feature space.

Methods: The Word2VisualVec Architecture

Word2VisualVec is a deep architecture designed to predict visual features from text through a multi-stage process involving multi-scale sentence vectorization and feature prediction.

The Word2VisualVec architecture for training and inference.

1. Multi-scale Sentence Vectorization

To capture both specific keywords and overall narrative structure, the model encodes a sentence using three parallel methods:

  • Bag-of-Words (BoW): Captures the presence of specific objects.
  • Word2Vec: Captures semantic similarity between words via mean pooling.
  • Gated Recurrent Unit (GRU): Captures sequential dependencies and long-term context.

2. Visual Feature Prediction

These three vectors are concatenated and fed through a Multi-Layer Perceptron (MLP). The MLP is trained to minimize the Mean Squared Error (MSE) between its output and the actual feature vector extracted from the image by a pre-trained ConvNet (e.g., ResNet-152).

3. Retrieval and Generalization

At test time, an image is converted to a feature vector. All candidate captions are passed through W2VV. Retrieval is performed by calculating the cosine similarity between the image vector and the predicted caption vectors. For video, the model is extended (Word2VideoVec) to predict 3D-CNN features (C3D) and audio features (MFCC).

Key Insights

  • Visual Space as the Gold Standard: A one-way mapping from text to visual space outperforms complex two-way joint embeddings, suggesting modern visual features are robust enough to serve as a universal anchor.
  • Multi-scale Representation: Combining BoW, Word2Vec, and GRU consistently outperforms any single method. BoW identifies "what" is in the image, while the GRU captures "how" the scene is described.
  • Vector Arithmetic: Because the model maps text to a vector space, it enables multi-modal querying. One can perform arithmetic on vectors (e.g., Image - "bicycle" + "motorbike") to find semantically related content.

Results and Performance

The model was evaluated on standard benchmarks including Flickr8k, Flickr30k, and the Microsoft Video Description (MSVD) dataset.

  • State-of-the-Art Image Retrieval: On Flickr30k, W2VV achieved a Recall@1 (R@1) of 45.9%, outperforming previous joint-subspace models like Order-embeddings (41.3%).
  • TrecVid Success: Word2VideoVec ranked 1st out of 21 submissions in the NIST TrecVid 2016 video caption retrieval task.
  • Scaling with Depth: Performance improves as the underlying ConvNet gets deeper (e.g., moving to ResNet-152), demonstrating that W2VV benefits from higher-quality visual features.

Conclusion

While W2VV shows significant promise, it remains dependent on the quality of pre-trained visual feature extractors. Future work may involve more sophisticated temporal modeling for video and using W2VV as a "prior" for generative captioning models.

Summary updated

Abstract

This paper strives to find amidst a set of sentences the one best describing the content of a given image or video. Different from existing works, which rely on a joint subspace for their image and video caption retrieval, we propose to do so in a visual space exclusively. Apart from this conceptual novelty, we contribute \emph{Word2VisualVec}, a deep neural network architecture that learns to predict a visual feature representation from textual input. Example captions are encoded into a textual embedding based on multi-scale sentence vectorization and further transferred into a deep visual feature of choice via a simple multi-layer perceptron. We further generalize Word2VisualVec for video caption retrieval, by predicting from text both 3-D convolutional neural network features as well as a visual-audio representation. Experiments on Flickr8k, Flickr30k, the Microsoft Video Description dataset and the very recent NIST TrecVid challenge for video caption retrieval detail Word2VisualVec's properties, its benefit over textual embeddings, the potential for multimodal query composition and its state-of-the-art results.
BibTeX
@article{dong2018predicting,
  author = {Jianfeng Dong and Xirong Li and Cees G. M. Snoek},
  title = {{Predicting Visual Features from Text for Image and Video Caption Retrieval}},
  journal = {IEEE Trans. Multimedia},
  year = {2018},
  url = {https://openreview.net/forum?id=nKt1pGbtTigl},
}
Figure