Abstract
This paper strives to find the sentence best describing the content of an image or video. Different from existing works, which rely on a joint subspace for image / video to sentence matching, we propose to do so in a visual space only. We contribute Word2VisualVec, a deep neural network architecture that learns to predict a deep visual encoding of textual input based on sentence vectorization and a multi-layer perceptron. We thoroughly analyze its architectural design, by varying the sentence vectorization strategy, network depth and the deep feature to predict for image to sentence matching. We also generalize Word2VisualVec for matching a video to a sentence, by extending the predictive abilities to 3-D ConvNet features as well as a visual-audio representation. Experiments on four challenging image and video benchmarks detail Word2VisualVec's properties, capabilities for image and video to sentence matching, and on all datasets its state-of-the-art results.
Introduction: Mapping Text to Vision
In multi-modal machine learning, the standard approach to matching images with descriptive text involves projecting both modalities into a shared "latent subspace." This middle ground allows for direct comparison but often introduces unnecessary complexity. The authors of Word2VisualVec propose a paradigm shift: instead of a shared space, why not map sentences directly into the high-dimensional visual feature space already established by state-of-the-art Convolutional Neural Networks (ConvNets)?
The core insight is that pre-trained models like ResNet-152 already provide a highly structured and rich representation of the visual world. By teaching a text model to "speak" this visual language, the system can leverage the discriminative power of visual features directly for cross-modal retrieval.
Methodology: The Word2VisualVec Architecture
The proposed architecture, Word2VisualVec, is a deep neural network designed to predict visual features from natural language. The process involves two primary stages: sentence vectorization and visual feature prediction.
Sentence Vectorization
To convert variable-length text into a fixed-length numerical input, three strategies were evaluated:
- Bag-of-Words (BoW): A simple count of word occurrences.
- Word Hashing: Decomposing words into letter-trigrams to improve robustness against rare words and misspellings.
- Word2Vec: Utilizing pre-trained word embeddings and mean-pooling them to capture broader semantic relationships.
Visual Feature Prediction
The resulting text vector is passed through a Multi-Layer Perceptron (MLP). Unlike traditional classifiers, this MLP is optimized to minimize the distance between its output and the actual deep feature vector of the corresponding image (e.g., a 2048-dimensional ResNet feature). The authors found that a 3-layer architecture (500-1000-2048) provides the optimal balance between mapping capacity and generalization.
For video content, the model is extended to Word2VideoVec. This version predicts a concatenated representation including 3D ConvNet features (for motion) and audio features (MFCCs), allowing the model to capture temporal and auditory context.
Experimental Evaluation and Results
The model was rigorously tested on four major benchmarks: Flickr8k and Flickr30k for image-to-sentence matching, and MSVD and the NIST TrecVid 2016 challenge for video-to-sentence matching.
State-of-the-Art Performance
Word2VisualVec achieved state-of-the-art results across these datasets. On Flickr30k, the model reached a Median Rank of 2, indicating that the ground-truth sentence is typically within the top two results. A key finding was that the choice of visual target matters: predicting ResNet-152 features significantly outperformed older architectures like AlexNet.
In the TrecVid 2016 Video-to-Text challenge, Word2VideoVec (incorporating both visual and audio features) outperformed 19 other submissions from international research teams, demonstrating the robustness of the visual-space mapping approach in complex video domains.
Conclusion and Future Directions
Word2VisualVec demonstrates that a dedicated joint subspace is not strictly necessary for effective cross-modal matching. By leveraging the existing structure of high-dimensional visual spaces, the model achieves high accuracy and extreme efficiency, with text-to-visual inference taking less than a millisecond.
Future work may involve replacing mean-pooling with more sophisticated sequence models like LSTMs or Transformers to better capture syntax in long sentences. Additionally, integrating further modalities such as depth or infrared data could further enhance the model's descriptive capabilities.
Summary updated