Video Corpus Moment Retrieval with Contrastive Learning

Hao Zhang, Aixin Sun, Wei Jing, Guoshun Nan, Liangli Zhen, Joey Tianyi Zhou, Rick Siow Mong Goh

Abstract

Video Corpus Moment Retrieval (VCMR) requires identifying specific temporal segments within a massive library of videos based on a natural language query. Traditionally, researchers have faced a stark trade-off between the speed of unimodal encoders and the accuracy of complex cross-modal interaction models. This paper introduces ReLoCLNet (Retrieval and Localization Network with Contrastive Learning), a framework that bridges this gap. By employing dual contrastive learning objectives—Video-level (VideoCL) and Frame-level (FrameCL)—the model aligns text and video features during training. This allows for high-accuracy retrieval during inference using simple, high-speed similarity calculations, achieving speeds up to 56 times faster than previous state-of-the-art methods.

Introduction

As digital video libraries expand exponentially, the ability to search for specific moments within thousands of hours of footage becomes critical. This task, known as Video Corpus Moment Retrieval (VCMR), involves two distinct challenges: identifying the relevant video from a large corpus and pinpointing the exact start and end times of the relevant action.

Existing solutions generally fall into two categories. Unimodal Encoding models process video and text separately, allowing for pre-computation and rapid retrieval but often sacrificing accuracy because the modalities do not interact until the final stage. Conversely, Cross-modal Interaction models feed video and text into heavy architectures that analyze fine-grained relationships. While accurate, these models are computationally expensive and often too slow for real-time applications, as they must process every video in the database against every new query.

The ReLoCLNet Framework

ReLoCLNet aims to provide the best of both worlds: the efficiency of unimodal architectures with the precision of cross-modal learning. The base architecture, ReLoNet, utilizes separate Transformer-based encoders for text queries and video content (including synchronized subtitles). To overcome the lack of explicit interaction layers, the researchers introduced a "simulated" interaction during the training phase via Contrastive Learning (CL).

Dual Contrastive Learning Objectives

The core innovation of ReLoCLNet lies in its two-pronged approach to feature alignment:

  1. Video-level Contrastive Learning (VideoCL): This objective ensures that the global representation of a text query is mathematically closer to its corresponding video than to any other video in the training batch. This optimizes the model's ability to select the correct video from the broader corpus.
  2. Frame-level Contrastive Learning (FrameCL): To address temporal localization, FrameCL operates within a single video. It treats frames within the target moment as "foreground" (positive) and frames outside that window as "background" (negative). By forcing the query representation to align specifically with foreground frames, the model learns to distinguish the exact moment of interest from the rest of the video.

By optimizing these objectives, the encoders learn to produce "pre-aligned" features. At search time, the model can use simple cosine similarity (late fusion) to match queries to moments, bypassing the need for heavy cross-modal computation.

Experimental Evaluation

The model was evaluated on two primary benchmarks:

  • TVR (TV Show Retrieval): A dataset containing 21.8K videos and 109K queries from popular TV shows, featuring synchronized subtitles.
  • ActivityNet Captions (ANetCaps): A standard benchmark for temporal video grounding involving approximately 20K YouTube videos.

Performance and Efficiency

ReLoCLNet demonstrated a transformative improvement in retrieval speed. On the TVR dataset, it achieved a 56.7x speedup over HAMMER, a leading cross-modal interaction model. Specifically, ReLoCLNet processed queries in approximately 3.8 milliseconds, whereas the competition required over 200 milliseconds.

In terms of accuracy, ReLoCLNet outperformed previous unimodal benchmarks and remained competitive with heavyweight cross-modal models. Ablation studies confirmed that the combination of VideoCL and FrameCL is essential; removing either objective resulted in a significant drop in performance, proving that alignment must occur at both the corpus and the temporal level.

Discussion and Future Work

While ReLoCLNet dominates in efficiency and broad retrieval metrics (such as Recall@10 or Recall@100), a slight performance gap remains in "Recall@1" at very strict Intersection over Union (IoU) thresholds compared to the most complex cross-modal models. This suggests that while contrastive learning is a powerful proxy for interaction, some extremely fine-grained nuances may still benefit from explicit cross-modal layers.

Future research directions include incorporating additional modalities such as audio features or Optical Character Recognition (OCR) for text appearing on-screen. Furthermore, applying large-scale video-text pre-training (similar to CLIP) could potentially close the remaining accuracy gap while maintaining the framework's signature efficiency.

Conclusion

ReLoCLNet represents a significant step forward for practical video search. By baking cross-modal intelligence directly into unimodal encoders through contrastive learning, it proves that high-performance video retrieval does not have to come at the cost of computational feasibility. For real-world applications like web search and surveillance, this sub-millisecond retrieval capability is a game-changer.

Summary updated

Abstract

Given a collection of untrimmed and unsegmented videos, video corpus moment retrieval (VCMR) is to retrieve a temporal moment (i.e., a fraction of a video) that semantically corresponds to a given text query. As video and text are from two distinct feature spaces, there are two general approaches to address VCMR: (i) to separately encode each modality representations, then align the two modality representations for query processing, and (ii) to adopt fine-grained cross-modal interaction to learn multi-modal representations for query processing. While the second approach often leads to better retrieval accuracy, the first approach is far more efficient. In this paper, we propose a Retrieval and Localization Network with Contrastive Learning (ReLoCLNet) for VCMR. We adopt the first approach and introduce two contrastive learning objectives to refine video encoder and text encoder to learn video and text representations separately but with better alignment for VCMR. The video contrastive learning (VideoCL) is to maximize mutual information between query and candidate video at video-level. The frame contrastive learning (FrameCL) aims to highlight the moment region corresponds to the query at frame-level, within a video. Experimental results show that, although ReLoCLNet encodes text and video separately for efficiency, its retrieval accuracy is comparable with baselines adopting cross-modal interaction learning.
BibTeX
@inproceedings{zhang2021video,
  author = {Hao Zhang and Aixin Sun and Wei Jing and Guoshun Nan and Liangli Zhen and Joey Tianyi Zhou and Rick Siow Mong Goh},
  title = {{Video Corpus Moment Retrieval with Contrastive Learning}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2021},
  url = {https://openreview.net/forum?id=7tX3uiJFZJR},
}
Figure