TL;DR
Instead of the traditional approach of mapping images and text into a shared "middle ground" (a joint subspace), this paper proposes Word2VisualVec (W2VV). W2VV maps natural language directly into a high-dimensional visual feature space, allowing for image and video retrieval to happen entirely on the "home turf" of computer vision.
What Problem Does This Paper Address?
The core challenge in cross-modal retrieval (finding the best caption for an image or vice versa) is the "semantic gap." Images are pixels; sentences are sequences of words. Historically, researchers handled this by learning a joint latent subspace where both modalities could be compared.
However, the authors argue that joint subspaces are unnecessarily complex. With the advent of powerful, pre-trained Convolutional Neural Networks (ConvNets) like ResNet and GoogLeNet, visual features have become highly discriminative and stable. The authors ask: Why not just teach the text model to "imagine" what the corresponding visual feature vector should look like?
Methods
The authors introduce Word2VisualVec (W2VV), a deep architecture designed to predict visual features from text.
- Multi-scale Sentence Vectorization: To capture both specific keywords and overall narrative structure, the model encodes a sentence using three parallel methods:
- Bag-of-Words (BoW): Captures the presence of specific objects.
- Word2Vec: Captures semantic similarity between words via mean pooling.
- Gated Recurrent Unit (GRU): Captures the sequential dependencies and long-term context of the sentence.
- Visual Feature Prediction: These three vectors are concatenated and fed through a Multi-Layer Perceptron (MLP). The MLP is trained to minimize the Mean Squared Error (MSE) between its output and the actual feature vector extracted from the image by a pre-trained ConvNet (e.g., ResNet-152).
- Word2VideoVec: For video, the model is extended to predict 3D-CNN features (C3D) and even audio features (MFCC), allowing the text to map into a multi-modal "video" space.
- Retrieval: At test time, an image is converted to a feature vector. All candidate captions are passed through W2VV. Retrieval is then a simple matter of calculating the cosine similarity between the image vector and the predicted caption vectors.
Datasets
The model was evaluated on several standard benchmarks:
- Image Retrieval: Flickr8k and Flickr30k.
- Video Retrieval: Microsoft Video Description (MSVD) dataset.
- Challenge Benchmark: The NIST TrecVid 2016 video caption retrieval task.
Key Insights and Ideas
- Visual Space as the Gold Standard: The most significant "aha moment" is that a one-way mapping from text to visual space outperforms complex two-way joint embeddings. This suggests that the visual features learned by modern ConvNets are robust enough to serve as a universal anchor for other modalities.
- Multi-scale is Mandatory: The authors found that using BoW, Word2Vec, and GRU together consistently beat any single method. BoW is great for "what" is in the image, while the GRU is better for "how" the scene is described (narrative).
- MSE vs. Ranking Loss: Interestingly, while many retrieval models use triplet or ranking losses, the authors found that simple Mean Squared Error (MSE) worked better for reconstructing the visual feature vector directly.
- Multi-modal Querying: Because the model maps text to a vector space, you can perform "vector arithmetic." For example, you can take the features of a "bicycle" image, subtract the predicted "bicycle" vector, and add a predicted "motorbike" vector to find images of motorbikes in similar scenes.
Key Results
- State-of-the-Art Performance: On Flickr30k, W2VV achieved a Recall@1 (R@1) of 45.9%, significantly outperforming previous joint-subspace models like Order-embeddings (41.3%) and Klein et al.’s Fisher Vector approach (35.0%).
- TrecVid Success: The model (Word2VideoVec) ranked 1st out of 21 submissions from 7 international teams in the TrecVid 2016 video caption retrieval task.
- Depth Matters: As the underlying ConvNet gets deeper (moving from CaffeNet to ResNet-152), the retrieval performance of W2VV improves, showing that the model scales with the quality of visual features.
Limitations and Future Work
- Dependence on Pre-trained Models: W2VV is only as good as the visual feature extractor it targets. If the ConvNet hasn't seen a specific object or action, W2VV cannot "predict" it effectively.
- Averaging Video Frames: For video, the authors used simple mean pooling over frames. They acknowledge that more sophisticated temporal modeling (like using LSTMs for the video side) could capture complex actions better.
- Fixed Sentence Pool: The current work focuses on retrieval from a pre-defined set of sentences. The authors suggest that W2VV could eventually be used as a "prior" or a guide for generative models that create entirely new captions.
Summary updated