Weakly Supervised Video Moment Retrieval From Text Queries.

Niluthpol Chowdhury Mithun, Sujoy Paul, Amit K. Roy-Chowdhury

Abstract

Recent methods in text-to-video moment retrieval using natural language queries typically require full supervision during training. However, acquiring temporal boundary annotations for large-scale video datasets is time-consuming and often unscalable. We introduce a weakly supervised framework for this task, where training relies only on video-level sentence descriptions without explicit temporal extents. Our approach utilizes a joint visual-semantic embedding framework and Text-Guided Attention (TGA) to discover latent alignments between video frames and text. Experiments on benchmark datasets show that our weakly supervised method achieves performance comparable to state-of-the-art fully supervised approaches.

Introduction

Video Moment Retrieval involves identifying the specific start and end times of an action within a long video based on a natural language query. While effective, traditional fully supervised models require expensive human-annotated temporal boundaries. This work explores a weakly supervised alternative, learning to localize moments using only the knowledge that a specific description applies to some part of a given video.

Examples of video moment retrieval where text queries are mapped to specific temporal segments.

Methodology: Text-Guided Attention

The proposed framework maps video and text features into a shared high-dimensional embedding space. The core innovation is Text-Guided Attention (TGA), which calculates the cosine similarity between the text query and individual video time steps.

  1. Feature Extraction: Visual features are extracted using pre-trained networks (C3D or VGG16), while text queries are encoded via a Gated Recurrent Unit (GRU).
  2. Latent Alignment: Similarity scores are processed through a softmax layer to generate an attention map, identifying relevant video segments.
  3. Training: The model is optimized using a hinge-based triplet ranking loss, ensuring that a video's correct description is closer to its visual features than mismatched descriptions.

During inference, the peaks in the TGA attention map serve as the predicted temporal boundaries for retrieval.

Experimental Evaluation

The model was tested on two benchmarks: Charades-STA (16,000+ pairs of indoor activities) and DiDeMo (27,000+ moments with a 5-second segment structure).

Key Results

On Charades-STA, the method achieved a Recall@1 (IoU 0.5) of 19.94%, outperforming visual-semantic baselines like VSA-STV (16.91%). On DiDeMo, the model reached an mIoU of 24.92, nearly matching the fully supervised LSTM-RGB-Local baseline (25.13). These results suggest that video-level labels provide sufficient signal for effective temporal localization.

Conclusion and Future Work

This research demonstrates that attention mechanisms can serve as both a weighting tool and a retrieval mechanism. By learning to satisfy a global ranking loss, the model implicitly learns to localize specific moments. Future improvements may involve incorporating audio cues or spatial-temporal modeling to better handle complex contexts and small objects.

Summary updated

Abstract

There have been a few recent methods proposed in text to video moment retrieval using natural language queries, but requiring full supervision during training. However, acquiring a large number of training videos with temporal boundary annotations for each text description is extremely time-consuming and often not scalable. In order to cope with this issue, in this work, we introduce the problem of learning from weak labels for the task of text to video moment retrieval. The weak nature of the supervision is because, during training, we only have access to the video-text pairs rather than the temporal extent of the video to which different text descriptions relate. We propose a joint visual-semantic embedding based framework that learns the notion of relevant segments from video using only video-level sentence descriptions. Specifically, our main idea is to utilize latent alignment between video frames and sentence descriptions using Text-Guided Attention (TGA). TGA is then used during the test phase to retrieve relevant moments. Experiments on two benchmark datasets demonstrate that our method achieves comparable performance to state-of-the-art fully supervised approaches.
BibTeX
@inproceedings{mithun2019weakly,
  author = {Niluthpol Chowdhury Mithun and Sujoy Paul and Amit K. Roy-Chowdhury},
  title = {{Weakly Supervised Video Moment Retrieval From Text Queries.}},
  booktitle = {CVPR},
  year = {2019},
  url = {https://openreview.net/forum?id=rQtVR6feu6S},
}
Figure