Weakly Supervised Video Moment Retrieval From Text Queries

Niluthpol Chowdhury Mithun, Sujoy Paul, Amit K. Roy-Chowdhury

Abstract

There have been a few recent methods proposed in text to video moment retrieval using natural language queries, but requiring full supervision during training. However, acquiring a large number of training videos with temporal boundary annotations for each text description is extremely time-consuming and often not scalable. In order to cope with this issue, in this work, we introduce the problem of learning from weak labels for the task of text to video moment retrieval. The weak nature of the supervision is because, during training, we only have access to the video-text pairs rather than the temporal extent of the video to which different text descriptions relate. We propose a joint visual-semantic embedding based framework that learns the notion of relevant segments from video using only video-level sentence descriptions. Specifically, our main idea is to utilize latent alignment between video frames and sentence descriptions using Text-Guided Attention (TGA). TGA is then used during the test phase to retrieve relevant moments. Experiments on two benchmark datasets demonstrate that our method achieves comparable performance to state-of-the-art fully supervised approaches.

Introduction

Text-to-video moment retrieval involves searching a long video for a specific segment that matches a natural language query, such as "a man holding a camera comes into view." While highly useful for video indexing and search, training models for this task traditionally requires full supervision. This means human annotators must manually mark the exact start and end timestamps for every text description across thousands of videos—a process that is tedious, expensive, and difficult to scale.

Examples of video moment retrieval queries and their corresponding temporal segments.

To address this bottleneck, we introduce a weakly supervised framework. Instead of requiring exact timestamps during training, our model only needs video-level descriptions (a video and a list of sentences describing its contents). The challenge lies in teaching the model to temporally localize an action when it has never explicitly been shown the temporal boundaries during training.

Methodology

The proposed framework utilizes a joint visual-semantic embedding to align video and text features. We extract features using standard pre-trained networks (C3D/VGG16 for video, GRU for text) and introduce a novel alignment module:

  • Text-Guided Attention (TGA): Since temporal boundaries are unknown during training, the model computes the cosine similarity between the text feature and every frame-level video feature. A softmax operation over the temporal dimension creates an attention map, highlighting frames most relevant to the text query.
  • Joint Embedding via Global Similarity: The TGA attention weights are used to pool temporal video features into a single, text-specific video vector.
  • Training: The network is trained using a hinge-based triplet ranking loss, minimizing the distance between the pooled video feature and its matching text feature while maximizing the distance to non-matching pairs.

During inference, the learned TGA attention weights are directly used to localize the segment that best matches the user's text query.

Experimental Setup

We evaluate our method on two benchmark datasets:

  • Charades-STA: 16,128 sentence-moment pairs built on the Charades dataset.
  • DiDeMo (Distinct Describable Moments): Over 26,000 annotated moments from diverse 30-second Flickr videos.

While these datasets provide ground-truth timestamps, we ignore them during training to simulate a weakly supervised setting, using them only for evaluation.

Results and Discussion

Our weakly supervised approach significantly outperforms baseline weakly supervised methods and remains competitive with fully supervised models.

Quantitative Performance

  • Charades-STA: At an IoU threshold of 0.5, the model achieved a Recall@1 of 17.04%, significantly higher than the weakly supervised baseline (VSA-RNN at 10.50%) and approaching the fully supervised CTRL model (23.63%).
  • DiDeMo: The model achieved a Recall@1 of 12.19% and a mean IoU of 24.92, nearly matching the fully supervised LSTM-RGB-Local baseline (13.10% R@1, 25.13 mIoU).

Key Insights

The success of this approach demonstrates that latent alignment is a powerful tool. By forcing the network to match a whole video to a specific sentence, the attention mechanism naturally learns to highlight the correct temporal segment. This allows for massive scalability, as models can be trained on easily scraped web datasets with video-level descriptions or auto-generated captions.

Limitations

The model currently struggles with small visual subjects (e.g., a small object mentioned in a query) and ambiguous spatial distinctions (e.g., standing in a doorway vs. entering a room). Future work will explore incorporating spatial attention, region proposals, and multimodal cues like audio to resolve these visual ambiguities.

Summary updated

Abstract

There have been a few recent methods proposed in text to video moment retrieval using natural language queries, but requiring full supervision during training. However, acquiring a large number of training videos with temporal boundary annotations for each text description is extremely time-consuming and often not scalable. In order to cope with this issue, in this work, we introduce the problem of learning from weak labels for the task of text to video moment retrieval. The weak nature of the supervision is because, during training, we only have access to the video-text pairs rather than the temporal extent of the video to which different text descriptions relate. We propose a joint visual-semantic embedding based framework that learns the notion of relevant segments from video using only video-level sentence descriptions. Specifically, our main idea is to utilize latent alignment between video frames and sentence descriptions using Text-Guided Attention (TGA). TGA is then used during the test phase to retrieve relevant moments. Experiments on two benchmark datasets demonstrate that our method achieves comparable performance to state-of-the-art fully supervised approaches.
BibTeX
@inproceedings{mithun2019weakly,
  author = {Niluthpol Chowdhury Mithun and Sujoy Paul and Amit K. Roy-Chowdhury},
  title = {{Weakly Supervised Video Moment Retrieval From Text Queries}},
  booktitle = {CVPR},
  year = {2019},
  url = {https://openreview.net/forum?id=rQtVR6feu6S},
}
Figure