TL;DR
Current video retrieval benchmarks suffer from a "needle-in-a-haystack" fallacy, assuming only one specific video matches a query despite many being semantically identical. This paper introduces Semantic Similarity Video Retrieval (SVR), a framework that uses automated NLP-based proxies to evaluate and train models based on how well they capture the true meaning of a scene, rather than just finding a single arbitrary ground-truth pair.
What Problem Does This Paper Address?
In standard video retrieval tasks, we evaluate models using Instance-based Video Retrieval (IVR). This assumes a strict 1:1 mapping: for every query caption, there is exactly one "correct" video in the dataset.
However, real-world datasets are messy. As shown in Figure 1, a query like "A man doing an origami tutorial" might have several perfectly valid matches in a large dataset. Under IVR, if a model retrieves a different, equally valid origami video instead of the one specific instance labeled as "ground truth," it is penalized. This makes current benchmarks "ad hoc" and potentially misleading, as they fail to reward models that understand the underlying semantics of the video.
Methods
The authors propose moving from binary (relevant/irrelevant) evaluation to a continuous Semantic Similarity scale.
1. Evaluation via nDCG
Instead of standard metrics like Recall@K or Median Rank, which require binary labels, the authors propose using normalized Discounted Cumulative Gain (nDCG). This metric accounts for varying levels of relevance, rewarding models for ranking highly similar items above less similar ones.
2. Semantic Similarity Proxies
Manually annotating millions of video-caption pairs for similarity is infeasible. To solve this, the authors propose four automated "proxies" that calculate similarity between captions as a stand-in for video-to-text similarity:
- Bag-of-Words (BoW): Simple word overlap (Intersection-over-Union).
- Part-of-Speech (PoS): Focuses on matching specific verbs and nouns to ensure actions and objects align.
- Synset-Aware (SYN): Uses WordNet to recognize that "stir" and "mix" are semantically related.
- METEOR (MET): A standard translation metric that considers both synonyms and sentence structure.
Figure 3 illustrates how these proxies assign different scores to "near-miss" captions, providing a much richer signal than a simple 0 or 1.
3. Training with Similarity
The authors also adapt the standard triplet loss for training. Instead of just picking a positive and a negative, they use the semantic proxies to define "relevance sets," allowing the model to learn from multiple related examples simultaneously.
Datasets
The analysis was performed on three major benchmarks:
- MSR-VTT: A large-scale open-domain dataset from YouTube.
- YouCook2: A cooking-specific dataset focusing on procedural tasks.
- EPIC-KITCHENS-100: A massive egocentric (first-person) dataset of daily kitchen activities.
Key Insights and Ideas
- The "Ground Truth" is a Bottleneck: Current models are often "unfairly penalized" because they find valid videos that aren't the official ground truth. Figure 5 shows that when you account for these "near-misses," the perceived performance of baseline models jumps significantly.
- Semantic Training Generalizes Better: Even if your end goal is standard IVR retrieval, training with semantic similarity proxies improves performance. It helps the model learn a more robust embedding space by treating similar items as "positives" rather than "hard negatives."
- Context vs. Semantics: Interestingly, the authors found that popular language models like BERT or Word2Vec are actually poor proxies for this task. They tend to relate words by co-occurrence (context) rather than direct meaning—for example, treating "open" and "close" as highly similar because they both appear near "door," even though they are opposite actions.
Key Results
- Performance Underestimation: The authors demonstrate that instance-based metrics can underestimate a model's true capability by a large margin (e.g., a ~30 point difference in Geometric Mean for EPIC-KITCHENS).
- Training Gains: Training the Collaborative Experts (CE) model with semantic knowledge consistently outperformed the instance-only baseline across all tested proxies. As seen in Figure 6, the improvements are most significant when the relevance threshold is tuned correctly (typically around $T=0.4$ for most proxies).
Limitations and Future Work
- Caption Dependency: The proxies rely entirely on the quality of the text captions. If a video contains information not captured in the text (like background music or specific lighting), these proxies cannot account for it.
- Proxy Selection: Different datasets benefit from different proxies (e.g., SYN is better for cooking where synonyms are common). Choosing the right proxy for a new domain remains a manual step.
- Future Directions: The authors suggest exploring more sophisticated loss functions beyond thresholded triplet loss and investigating how to incorporate audio-visual cues directly into the similarity proxies.
Summary updated