Word2VisualVec: Cross-Media Retrieval by Visual Feature Prediction

Jianfeng Dong, Xirong Li, Cees G. M. Snoek

Abstract

This paper strives to find the sentence best describing the content of an image or video. Different from existing works, which rely on a joint subspace for image / video to sentence matching, we propose to do so in a visual space only. We contribute Word2VisualVec, a deep neural network architecture that learns to predict a deep visual encoding of textual input based on sentence vectorization and a multi-layer perceptron. We thoroughly analyze its architectural design, by varying the sentence vectorization strategy, network depth and the deep feature to predict for image to sentence matching. We also generalize Word2VisualVec for matching a video to a sentence, by extending the predictive abilities to 3-D ConvNet features as well as a visual-audio representation. Experiments on four challenging image and video benchmarks detail Word2VisualVec's properties, capabilities for image and video to sentence matching, and on all datasets its state-of-the-art results.

Introduction

Cross-modal retrieval, such as matching a descriptive sentence to an image or video, typically involves bridging the semantic gap between visual pixels and linguistic tokens. Conventional approaches often map both modalities into a shared, joint latent subspace. However, this paper argues that modern Convolutional Neural Networks (CNNs), like ResNet, already provide a highly optimized and semantically rich visual feature space. Instead of learning a new neutral ground, the authors propose Word2VisualVec, an architecture designed to predict these deep visual features directly from natural language. By treating the pre-trained CNN's feature space as the ground truth, the model learns to "imagine" the visual representation of a sentence.

Methodology

The Word2VisualVec architecture follows a three-stage pipeline to map text into the visual domain:

  1. Sentence Vectorization: Natural language input is converted into a fixed-length vector. The authors evaluate three strategies: Bag-of-Words (BoW), Word Hashing (using letter-trigrams to handle out-of-vocabulary terms), and pre-trained word2vec embeddings (aggregated via mean-pooling).
  2. Multi-Layer Perceptron (MLP): The vectorized text is processed through a deep MLP, typically consisting of three layers. This network learns the non-linear transformation required to map linguistic features into the high-dimensional visual space.
  3. Visual Feature Prediction: The MLP's output is trained to match the features extracted by a pre-trained CNN (e.g., the pool5 layer of ResNet-152) from the corresponding image or video.
The Word2VisualVec architecture for training and inference, illustrating the mapping from sentence vectorization to visual feature prediction.

The model is optimized using Mean Squared Error (MSE) to minimize the distance between the predicted text-based feature and the actual visual feature. For video retrieval, the approach is extended to predict 3D ConvNet (C3D) features and can incorporate audio representations like MFCCs.

Experimental Results

The model was evaluated on standard image benchmarks (Flickr8k, Flickr30k) and video datasets (MSVD, NIST TrecVid 2016).

Image and Video Matching

On Flickr8k, Word2VisualVec achieved a Recall@1 of 33.6%, outperforming more complex multimodal Recurrent Neural Networks. A key insight is that as the underlying CNN anchor becomes more powerful (e.g., moving from AlexNet to ResNet-152), the retrieval performance improves significantly. Furthermore, the model naturally handles synonymy; for instance, "a dog leaps" and "a dog is jumping" are mapped to similar regions in the visual feature space because they share a common visual ground truth.

TrecVid 2016 Challenge

The "Word2VideoVec" variant, which incorporates temporal and audio features, was submitted to the NIST TrecVid 2016 Video-to-Text task. It achieved top-tier performance, outperforming 19 other runs from international teams.

Mean inverted rank results on TrecVid 2016 sets A and B, showing Word2VideoVec (with and without audio) outperforming other submissions.

Conclusion

Word2VisualVec demonstrates that a dedicated visual feature space is a robust and efficient medium for cross-modal retrieval. By bypassing the need for a joint latent subspace and leveraging the power of pre-trained CNNs, the model achieves state-of-the-art results with high computational efficiency (0.86 ms per sentence). Future work may involve replacing mean-pooling with sequence models like LSTMs to better capture complex sentence structures and further exploring multi-modal integration in datasets with rich audio-visual content.

Summary updated

Abstract

This paper strives to find the sentence best describing the content of an image or video. Different from existing works, which rely on a joint subspace for image / video to sentence matching, we propose to do so in a visual space only. We contribute Word2VisualVec, a deep neural network architecture that learns to predict a deep visual encoding of textual input based on sentence vectorization and a multi-layer perceptron. We thoroughly analyze its architectural design, by varying the sentence vectorization strategy, network depth and the deep feature to predict for image to sentence matching. We also generalize Word2VisualVec for matching a video to a sentence, by extending the predictive abilities to 3-D ConvNet features as well as a visual-audio representation. Experiments on four challenging image and video benchmarks detail Word2VisualVec's properties, capabilities for image and video to sentence matching, and on all datasets its state-of-the-art results.
BibTeX
@misc{dong2016word,
  author = {Jianfeng Dong and Xirong Li and Cees G. M. Snoek},
  title = {{Word2VisualVec: Cross-Media Retrieval by Visual Feature Prediction}},
  year = {2016},
  eprint = {1604.06838},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1604.06838},
}
Figure