Abstract
Traditional vision-language models often collapse diverse textual descriptions of a single image into a single point in the embedding space. This approach ignores the inherent multi-faceted nature of complex scenes, where different observers may focus on distinct semantic elements. We present Multi-View Visual Semantic Embedding (MV-VSE), a framework designed to learn multiple distinct embeddings for each image. By utilizing a novel multi-view architecture and an "upper bound" triplet loss, MV-VSE ensures that diverse perspectives are captured and preserved. Our method achieves state-of-the-art performance on standard image-text retrieval benchmarks, particularly in datasets with high intra-class variation.
Introduction: The Problem of Intra-Class Variation
In cross-modal retrieval, the objective is to project images and text into a shared mathematical space where matching pairs are positioned closely together. This is typically achieved through Visual Semantic Embedding (VSE). However, the authors identify a fundamental flaw in standard VSE: intra-class variation.
A single image in datasets like Flickr30K or MS-COCO is often associated with five different captions. As shown in the figure above, one caption might focus on "people walking on the street," while another highlights a "man at a hotdog stand." Standard models attempt to force a single image embedding to align with all these diverse descriptions simultaneously. This "averaging" effect suppresses unique details, reducing the model's flexibility and its ability to generalize to nuanced queries.
The MV-VSE Framework
To address this, the authors propose the Multi-View Visual Semantic Embedding (MV-VSE) framework. Instead of a single vector, the model generates $K$ distinct "views" for every image.
Multi-View Architecture
The framework employs a shared feature extractor (such as ResNet or Faster R-CNN) to derive global features. These features are then passed through $K$ independent "aggregators"—small neural network layers—that each produce a unique visual embedding. This allows the model to represent different semantic regions or concepts within the same image independently.
Inference and Optimization
During retrieval, the model calculates the similarity between a text query and all $K$ visual views, selecting the maximum score as the final match. This ensures that if any single internal "view" aligns with the user's query, the retrieval is successful.
The core innovation in training is the Multi-View Upper Bound Loss ($L_{MV-Up-Tri}$). Standard triplet losses often fail in multi-view settings by either optimizing only the best view (leaving others untrained) or averaging all views (destroying diversity). The proposed loss acts as a mathematical upper bound that optimizes all branches simultaneously but ceases to push a specific view once it satisfies the matching criteria. This mechanism forces different branches to specialize in different aspects of the image.
Experimental Results and Insights
MV-VSE was evaluated on the Flickr30K and MS-COCO benchmarks. The model achieved state-of-the-art results, notably improving the RSUM (Recall Sum) on Flickr30K from 513.5 to 517.5 when using a BERT-based backbone.
Key findings from the study include:
- Semantic Diversity: Using Sentence-BERT, the authors quantified that captions for the same image often have semantic similarities as low as 0.4, justifying the need for multiple embeddings.
- Efficiency: MV-VSE is a "plug-and-play" wrapper. Adding extra views typically adds fewer than 0.1M parameters, representing less than 1% overhead compared to the total model size.
- Optimal View Count: Ablation studies indicated that $K=3$ views provide the best balance between capturing nuance and avoiding redundancy for current benchmarks.
Conclusion and Future Directions
MV-VSE effectively captures the multi-faceted nature of visual scenes by allowing for specialized embedding branches. While the current implementation uses a fixed number of views ($K$), future work may explore dynamic view allocation based on image complexity. Additionally, the authors suggest that extending this multi-view logic to textual embeddings and video-text retrieval could yield further performance gains in capturing complex, time-varying perspectives.
Summary updated