Abstract
Video Corpus Moment Retrieval (VCMR) requires identifying specific temporal segments within a massive library of videos based on a natural language query. Traditionally, researchers have faced a stark trade-off between the speed of unimodal encoders and the accuracy of complex cross-modal interaction models. This paper introduces ReLoCLNet (Retrieval and Localization Network with Contrastive Learning), a framework that bridges this gap. By employing dual contrastive learning objectives—Video-level (VideoCL) and Frame-level (FrameCL)—the model aligns text and video features during training. This allows for high-accuracy retrieval during inference using simple, high-speed similarity calculations, achieving speeds up to 56 times faster than previous state-of-the-art methods.
Introduction
As digital video libraries expand exponentially, the ability to search for specific moments within thousands of hours of footage becomes critical. This task, known as Video Corpus Moment Retrieval (VCMR), involves two distinct challenges: identifying the relevant video from a large corpus and pinpointing the exact start and end times of the relevant action.
Existing solutions generally fall into two categories. Unimodal Encoding models process video and text separately, allowing for pre-computation and rapid retrieval but often sacrificing accuracy because the modalities do not interact until the final stage. Conversely, Cross-modal Interaction models feed video and text into heavy architectures that analyze fine-grained relationships. While accurate, these models are computationally expensive and often too slow for real-time applications, as they must process every video in the database against every new query.
The ReLoCLNet Framework
ReLoCLNet aims to provide the best of both worlds: the efficiency of unimodal architectures with the precision of cross-modal learning. The base architecture, ReLoNet, utilizes separate Transformer-based encoders for text queries and video content (including synchronized subtitles). To overcome the lack of explicit interaction layers, the researchers introduced a "simulated" interaction during the training phase via Contrastive Learning (CL).
Dual Contrastive Learning Objectives
The core innovation of ReLoCLNet lies in its two-pronged approach to feature alignment:
- Video-level Contrastive Learning (VideoCL): This objective ensures that the global representation of a text query is mathematically closer to its corresponding video than to any other video in the training batch. This optimizes the model's ability to select the correct video from the broader corpus.
- Frame-level Contrastive Learning (FrameCL): To address temporal localization, FrameCL operates within a single video. It treats frames within the target moment as "foreground" (positive) and frames outside that window as "background" (negative). By forcing the query representation to align specifically with foreground frames, the model learns to distinguish the exact moment of interest from the rest of the video.
By optimizing these objectives, the encoders learn to produce "pre-aligned" features. At search time, the model can use simple cosine similarity (late fusion) to match queries to moments, bypassing the need for heavy cross-modal computation.
Experimental Evaluation
The model was evaluated on two primary benchmarks:
- TVR (TV Show Retrieval): A dataset containing 21.8K videos and 109K queries from popular TV shows, featuring synchronized subtitles.
- ActivityNet Captions (ANetCaps): A standard benchmark for temporal video grounding involving approximately 20K YouTube videos.
Performance and Efficiency
ReLoCLNet demonstrated a transformative improvement in retrieval speed. On the TVR dataset, it achieved a 56.7x speedup over HAMMER, a leading cross-modal interaction model. Specifically, ReLoCLNet processed queries in approximately 3.8 milliseconds, whereas the competition required over 200 milliseconds.
In terms of accuracy, ReLoCLNet outperformed previous unimodal benchmarks and remained competitive with heavyweight cross-modal models. Ablation studies confirmed that the combination of VideoCL and FrameCL is essential; removing either objective resulted in a significant drop in performance, proving that alignment must occur at both the corpus and the temporal level.
Discussion and Future Work
While ReLoCLNet dominates in efficiency and broad retrieval metrics (such as Recall@10 or Recall@100), a slight performance gap remains in "Recall@1" at very strict Intersection over Union (IoU) thresholds compared to the most complex cross-modal models. This suggests that while contrastive learning is a powerful proxy for interaction, some extremely fine-grained nuances may still benefit from explicit cross-modal layers.
Future research directions include incorporating additional modalities such as audio features or Optical Character Recognition (OCR) for text appearing on-screen. Furthermore, applying large-scale video-text pre-training (similar to CLIP) could potentially close the remaining accuracy gap while maintaining the framework's signature efficiency.
Conclusion
ReLoCLNet represents a significant step forward for practical video search. By baking cross-modal intelligence directly into unimodal encoders through contrastive learning, it proves that high-performance video retrieval does not have to come at the cost of computational feasibility. For real-world applications like web search and surveillance, this sub-millisecond retrieval capability is a game-changer.
Summary updated