On Semantic Similarity in Video Retrieval

Michael Wray, Hazel Doughty, Dima Damen

TL;DR

Current video retrieval benchmarks suffer from a "needle-in-a-haystack" fallacy, assuming only one specific video matches a query despite many being semantically identical. This paper introduces Semantic Similarity Video Retrieval (SVR), a framework that uses automated NLP-based proxies to evaluate and train models based on how well they capture the true meaning of a scene, rather than just finding a single arbitrary ground-truth pair.

What Problem Does This Paper Address?

In standard video retrieval tasks, we evaluate models using Instance-based Video Retrieval (IVR). This assumes a strict 1:1 mapping: for every query caption, there is exactly one "correct" video in the dataset.

However, real-world datasets are messy. As shown in Figure 1, a query like "A man doing an origami tutorial" might have several perfectly valid matches in a large dataset. Under IVR, if a model retrieves a different, equally valid origami video instead of the one specific instance labeled as "ground truth," it is penalized. This makes current benchmarks "ad hoc" and potentially misleading, as they fail to reward models that understand the underlying semantics of the video.

Figure 1: All current video retrieval works treat caption collected for a certain video as relevant, even when other videos are equally relevant to a query text. This makes the evaluation of popular datasets ad hoc at times. We propose to use continuous similarity, allowing multiple videos to be treated as equally relevant. Ex. from MSR-VTT [66].

Methods

The authors propose moving from binary (relevant/irrelevant) evaluation to a continuous Semantic Similarity scale.

1. Evaluation via nDCG

Instead of standard metrics like Recall@K or Median Rank, which require binary labels, the authors propose using normalized Discounted Cumulative Gain (nDCG). This metric accounts for varying levels of relevance, rewarding models for ranking highly similar items above less similar ones.

2. Semantic Similarity Proxies

Manually annotating millions of video-caption pairs for similarity is infeasible. To solve this, the authors propose four automated "proxies" that calculate similarity between captions as a stand-in for video-to-text similarity:

  • Bag-of-Words (BoW): Simple word overlap (Intersection-over-Union).
  • Part-of-Speech (PoS): Focuses on matching specific verbs and nouns to ensure actions and objects align.
  • Synset-Aware (SYN): Uses WordNet to recognize that "stir" and "mix" are semantically related.
  • METEOR (MET): A standard translation metric that considers both synonyms and sentence structure.

Figure 3 illustrates how these proxies assign different scores to "near-miss" captions, providing a much richer signal than a simple 0 or 1.

Proxy measure examples

3. Training with Similarity

The authors also adapt the standard triplet loss for training. Instead of just picking a positive and a negative, they use the semantic proxies to define "relevance sets," allowing the model to learn from multiple related examples simultaneously.

Datasets

The analysis was performed on three major benchmarks:

  • MSR-VTT: A large-scale open-domain dataset from YouTube.
  • YouCook2: A cooking-specific dataset focusing on procedural tasks.
  • EPIC-KITCHENS-100: A massive egocentric (first-person) dataset of daily kitchen activities.

Key Insights and Ideas

  • The "Ground Truth" is a Bottleneck: Current models are often "unfairly penalized" because they find valid videos that aren't the official ground truth. Figure 5 shows that when you account for these "near-misses," the perceived performance of baseline models jumps significantly.
  • Semantic Training Generalizes Better: Even if your end goal is standard IVR retrieval, training with semantic similarity proxies improves performance. It helps the model learn a more robust embedding space by treating similar items as "positives" rather than "hard negatives."
  • Context vs. Semantics: Interestingly, the authors found that popular language models like BERT or Word2Vec are actually poor proxies for this task. They tend to relate words by co-occurrence (context) rather than direct meaning—for example, treating "open" and "close" as highly similar because they both appear near "door," even though they are opposite actions.
Performance bounds under semantic similarity

Key Results

  • Performance Underestimation: The authors demonstrate that instance-based metrics can underestimate a model's true capability by a large margin (e.g., a ~30 point difference in Geometric Mean for EPIC-KITCHENS).
  • Training Gains: Training the Collaborative Experts (CE) model with semantic knowledge consistently outperformed the instance-only baseline across all tested proxies. As seen in Figure 6, the improvements are most significant when the relevance threshold is tuned correctly (typically around $T=0.4$ for most proxies).
Training with semantic knowledge vs instance-only

Limitations and Future Work

  • Caption Dependency: The proxies rely entirely on the quality of the text captions. If a video contains information not captured in the text (like background music or specific lighting), these proxies cannot account for it.
  • Proxy Selection: Different datasets benefit from different proxies (e.g., SYN is better for cooking where synonyms are common). Choosing the right proxy for a new domain remains a manual step.
  • Future Directions: The authors suggest exploring more sophisticated loss functions beyond thresholded triplet loss and investigating how to incorporate audio-visual cues directly into the similarity proxies.

Summary updated

Abstract

Current video retrieval efforts all found their evaluation on an instance-based assumption, that only a single caption is relevant to a query video and vice versa. We demonstrate that this assumption results in performance comparisons often not indicative of models' retrieval capabilities. We propose a move to semantic similarity video retrieval, where (i) multiple videos/captions can be deemed equally relevant, and their relative ranking does not affect a method's reported performance and (ii) retrieved videos/captions are ranked by their similarity to a query. We propose several proxies to estimate semantic similarities in large-scale retrieval datasets, without additional annotations. Our analysis is performed on three commonly used video retrieval datasets (MSR-VTT, YouCook2 and EPIC-KITCHENS).
BibTeX
@inproceedings{wray2021semantic,
  author = {Michael Wray and Hazel Doughty and Dima Damen},
  title = {{On Semantic Similarity in Video Retrieval}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=n2fHCRjB-cE},
}
Figure