Abstract
The goal of weakly-supervised video moment retrieval is to localize the video segment most relevant to the given natural language query without access to temporal annotations during training. Prior strongly- and weakly-supervised approaches often leverage co-attention mechanisms to learn visual-semantic representations for localization. However, while such approaches tend to focus on identifying relationships between elements of the video and language modalities, there is less emphasis on modeling relational context between video frames given the semantic context of the query. Consequently, the above-mentioned visual-semantic representations, built upon local frame features, do not contain much contextual information. To address this limitation, we propose a Latent Graph Co-Attention Network (LoGAN) that exploits fine-grained frame-by-word interactions to reason about correspondences between all possible pairs of frames, given the semantic context of the query. Comprehensive experiments across two datasets, DiDeMo and Charades-Sta, demonstrate the effectiveness of our proposed latent co-attention model where it outperforms current state-of-the-art (SOTA) weakly-supervised approaches by a significant margin. Notably, it even achieves a 11% improvement to Recall@1 accuracy over strongly-supervised SOTA methods on DiDeMo.
Introduction
Video moment retrieval aims to identify the precise start and end times of an event within a longer video based on a natural language query (e.g., "a person sits down on a chair"). While strongly supervised models achieve high performance, they rely on expensive human-annotated timestamps. Weakly-supervised methods attempt to learn this mapping using only video-sentence pairs.
However, traditional weakly-supervised approaches often treat video frames as independent entities or rely on global sentence representations. This ignores the relational context—how a frame's significance is defined by its neighbors and specific words in the query. For example, a frame showing a baby only gains specific relevance for the query "man talking to a baby" when viewed in relation to adjacent frames containing the man.
The LoGAN Framework
To overcome these limitations, the authors introduce the Latent Graph Co-Attention Network (LoGAN). Unlike previous models that use global representations, LoGAN focuses on fine-grained, iterative reasoning through three core components:
- Frame-By-Word (FBW) Interactions: LoGAN computes similarity scores between every individual word and every video frame. This allows the model to anchor specific keywords (like "holding" or "glass") to relevant visual segments.
- Word-Conditioned Visual Graph (WCVG): This core innovation treats video frames as nodes in a graph. Through iterative message-passing, each frame updates its representation by aggregating information from other frames, conditioned on the query's semantics. This enables the model to capture temporal flow and relational context.
- Positional Encodings (PE): Inspired by Transformers, the model incorporates sine and cosine-based positional encodings into video features, providing a sense of sequence and time without explicit labels.
The system is trained using a Multiple Instance Learning (MIL) paradigm with a margin-based ranking loss, ensuring that positive video-query pairs are scored higher than "hard negative" pairs.
Key Insights and Results
LoGAN demonstrates that contextualization is crucial; a frame's relevance is often latent and only becomes clear when analyzed alongside neighboring frames through the lens of specific query words. Furthermore, the results show that a well-designed weakly-supervised model can outperform strongly-supervised counterparts by effectively modeling internal video and language structures.
Performance Benchmarks
LoGAN achieved state-of-the-art (SOTA) results on two major benchmarks:
- DiDeMo: Achieved 39.20% Recall@1, nearly tripling the previous weakly-supervised SOTA (TGA at 12.19%) and outperforming strongly-supervised methods (like TGN and MCN) by approximately 11%.
- Charades-STA: At an IoU threshold of 0.5, LoGAN reached 34.68% Recall@1, doubling the performance of the TGA baseline (17.04%).
Ablation studies confirmed that both the graph-based message-passing and positional encodings were vital to these performance gains.
Limitations and Future Work
While LoGAN excels at identifying the primary segment (Recall@1), its Recall@5 on DiDeMo is lower than some strongly-supervised methods, suggesting it may struggle to rank secondary candidate segments. Additionally, the current reliance on pre-defined sliding windows could be improved with a learnable proposal mechanism. Finally, the computational complexity of iterative graph message-passing remains a consideration for real-time deployment.
Summary updated