Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video

Zhenfang Chen, Lin Ma, Wenhan Luo, Kwan-Yee K. Wong

TL;DR

This paper introduces a new task: weakly-supervised spatio-temporal grounding of natural sentences in videos. They propose a novel "attentive interactor" model that learns to pinpoint the relevant video segment (a spatio-temporal tube) corresponding to a sentence, without needing detailed frame-by-frame annotations. They also release a new dataset, VID-sentence, to facilitate research in this area.

What Problem Does This Paper Address?

The goal of "grounding" is to connect language descriptions to specific regions in images or videos. While image grounding has seen progress, video grounding is more challenging due to the temporal dimension. Existing methods often require dense, frame-level annotations, which are extremely labor-intensive to create. Even "weakly-supervised" approaches typically focus on grounding simple nouns or pronouns in static frames, which is insufficient for understanding complex sentences that describe actions or events unfolding over time. For example, simply identifying "dog" in a frame doesn't capture the action of "lying on the grass and then it stands up." This paper tackles the problem of grounding full, natural sentences in videos using only video-sentence pairs, avoiding the need for precise spatio-temporal tube annotations during training.

Methods

The proposed approach, Weakly-Supervised Spatio-Temporally Grounding (WSSTG), frames the problem as a multiple instance learning (MIL) task.

  1. Instance Generation: They first extract candidate "spatio-temporal tubes" (sequences of bounding boxes across frames) from the video. This is done by using a region proposal network (similar to Faster-RCNN) to detect bounding boxes in each frame and then linking them using a score based on confidence and overlap. This process is repeated until no more bounding boxes can be linked, generating a set of instance proposals.
  2. Attentive Interactor: This is the core novel component. It takes the visual features of an instance proposal and the sentence features as input.
  3. Interaction Module: It uses LSTMs to encode both the visual (instance proposal) and textual (sentence) sequences. Then, a fine-grained interaction is achieved using an attention mechanism. For each visual feature segment, it adaptively summarizes the sentence features, creating "visually guided sentence features." This allows the model to focus on sentence words that are most relevant to the current visual segment.
  4. Matching Behavior Characterization: This module takes the interaction outputs and computes a matching score between the instance proposal and the sentence. This score represents how well the visual tube semantically aligns with the language query.
  5. Training: The model is trained using a combination of losses:
  6. Ranking Loss: This standard MIL loss aims to ensure that the matching score for an aligned video-sentence pair is higher than for unaligned pairs.
  7. Diversity Loss: This novel loss encourages a diverse distribution of matching scores across all instance proposals from a given video. It helps to strengthen the scores for truly relevant instances and penalize those that are less relevant, even within an aligned video-sentence pair.

Datasets

  • VID-sentence Dataset: The authors introduce this new dataset, built upon the ImageNet Video Object Detection (VID) dataset. It consists of sentence descriptions for spatio-temporal instances extracted from videos. They selected 30 categories from the VID dataset and annotated 7,654 instances with natural language descriptions. This dataset is designed to be a benchmark for the WSSTG task, offering more challenging and diverse content than previous datasets like Person-sentence.
  • Person-sentence Dataset: Used for additional evaluation, this dataset focuses on grounding sentences related to people in videos.

Key Insights and Ideas

  • Grounding full sentences is crucial: Simply grounding nouns is insufficient for understanding complex video content described by natural language. The temporal dynamics and actions are key.
  • Attention for fine-grained alignment: The "attentive interactor" effectively bridges the gap between visual and textual modalities by allowing the model to dynamically focus on relevant words in the sentence for each part of the video instance.
  • Diversity loss for better discrimination: The novel diversity loss is critical for WSSTG. It helps the model distinguish truly relevant spatio-temporal tubes from other plausible but incorrect ones within the same video, especially when dealing with many instance proposals.
  • VID-sentence dataset is a significant contribution: The lack of suitable datasets was a major bottleneck. VID-sentence provides a more challenging and comprehensive benchmark for this specific task.

Key Results

  • The proposed WSSTG model significantly outperforms several baseline methods on the VID-sentence dataset, achieving an average accuracy of 37.2% (at IoU threshold 0.4, 0.5, 0.6).
  • The attentive interactor and diversity loss both contribute positively to performance, with the full model achieving the best results.
  • Qualitative results show the model can identify more accurate and temporally consistent instances compared to baselines.
  • On the Person-sentence dataset, the model also shows strong performance, achieving 51.0% accuracy, and the authors note that the VID-sentence dataset is more challenging.

Limitations and Future Work

  • Dataset Size and Complexity: While VID-sentence is an improvement, the authors acknowledge that the upper bound performance on this dataset (47.6%) is lower than on the Person-sentence dataset (77.9%), suggesting it's more challenging. Further work could explore larger and more diverse datasets.
  • Instance Proposal Generation: The current method relies on off-the-shelf region proposal networks. Future work could explore more sophisticated instance generation techniques tailored for spatio-temporal tubes.
  • Generalization: While the model shows good generalization to the Person-sentence dataset, exploring its performance on even more diverse video types and sentence structures would be valuable.
  • Computational Cost: The paper mentions the complexity of generating and processing many instance proposals, which could be an area for optimization.

Summary updated

Abstract

In this paper, we address a novel task, namely weakly-supervised spatio-temporally grounding natural sentence in video. Specifically, given a natural sentence and a video, we localize a spatio-temporal tube in the video that semantically corresponds to the given sentence, with no reliance on any spatio-temporal annotations during training. First, a set of spatio-temporal tubes, referred to as instances, are extracted from the video. We then encode these instances and the sentence using our proposed attentive interactor which can exploit their fine-grained relationships to characterize their matching behaviors. Besides a ranking loss, a novel diversity loss is introduced to train the proposed attentive interactor to strengthen the matching behaviors of reliable instance-sentence pairs and penalize the unreliable ones. Moreover, we also contribute a dataset, called VID-sentence, based on the ImageNet video object detection dataset, to serve as a benchmark for our task. Extensive experimental results demonstrate the superiority of our model over the baseline approaches.
BibTeX
@inproceedings{chen2019weakly,
  author = {Zhenfang Chen and Lin Ma and Wenhan Luo and Kwan-Yee K. Wong},
  title = {{Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1183},
}
Figure