Not All Frames Are Equal: Weakly-Supervised Video Grounding With Contextual Similarity and Visual Clustering Losses

Jing Shi, Jia Xu, Boqing Gong, Chenliang XuCVPR 2019 · OpenReview

Abstract

We investigate the problem of weakly-supervised video grounding, where only video-level sentences are provided. This is a challenging task, as previous Multi-Instance Learning (MIL) based image grounding methods tend to fail in the video domain. Recent work attempts to decompose video-level MIL into frame-level MIL by applying weighted sentence-frame ranking loss over frames, but it is not robust and does not exploit the rich temporal information in videos. In this work, we address these issues by extending frame-level MIL with a false positive frame-bag constraint and modeling the visual feature consistency in the video. Specifically, we design a contextual similarity between semantic and visual features to deal with sparse object association across frames. Furthermore, we leverage temporal coherence by strengthening the clustering effect of similar features in the visual space. We conduct an extensive evaluation on YouCookII and RoboWatch datasets, demonstrating that our method significantly outperforms prior state-of-the-art methods.

Introduction

Visual grounding is the task of localizing a text query (e.g., "the potato") within an image or video. In a weakly-supervised setting, the model only sees video-level descriptions during training, never explicit (x, y) coordinates or bounding boxes. This makes the task significantly more scalable but also much harder to optimize.

While this works reasonably well for static images using Multiple Instance Learning (MIL), videos introduce a unique "dilemma." If you treat the entire video as one "bag" of regions, the bag becomes too large for the model to learn effectively—the signal of the correct object is lost among thousands of irrelevant regions. Conversely, if you treat every frame as a "bag" (frame-level MIL), you run into the problem of false positives: a sentence might mention a "pot," but the pot might only appear in 50% of the video frames. Standard models get confused by trying to find the object in frames where it doesn't exist.

Visual grounding aims to localize text queries within video frames using only video-level supervision.

Methodology

The proposed approach introduces two main innovations to solve the MIL dilemma by filtering noise and leveraging temporal consistency.

Contextual Similarity

Instead of treating every frame as equally important, the model looks at the entire video segment to calculate a "contextual gain." It performs a 0-1 normalization of similarity scores across all frames. This effectively "down-weights" frames where the query object is likely missing and "up-weights" the key frames where the object is most prominent. This acts as a self-correcting mechanism to ignore the noise inherent in video-level labels.

Visual Clustering Loss

This leverages the temporal nature of video. The intuition is simple: if the model thinks a region in Frame A is a "potato" and a region in Frame B is also a "potato," those two regions should look similar in the feature space. By minimizing the negative cosine similarity between these regions, the model learns a more robust and discriminative visual embedding. In video, the fact that an object persists over time is a "free" supervisory signal; the clustering loss exploits this without needing extra labels.

The MIL framework for video grounding organizes regions into bags associated with video-level labels, where contextual similarity helps identify the most relevant frames.

Experimental Results

The researchers evaluated their approach on two primary benchmarks: YouCookII (2,000 challenging cooking videos) and RoboWatch (used to test generalization across different domains).

  • State-of-the-Art Performance: On the YouCookII dataset, the proposed method (VisClus+CtxSim) achieved a Box Accuracy of 40.71%, significantly outperforming the previous state-of-the-art which sat at 35.08%.
  • Ablation Success: Both components were critical. Contextual similarity alone improved accuracy, but adding visual clustering provided a further boost, proving that temporal consistency is a vital cue.
  • Generalization: On the RoboWatch dataset, the model outperformed the frame-level MIL baseline by 3.43%. This demonstrates that the model can generalize to unseen video classes and "zero-shot" queries (like "hanger") by transferring knowledge from similar visual features encountered during training.

Conclusion and Future Work

The core philosophy of this work is that "Not All Frames Are Equal." By acknowledging that many frames in a video are irrelevant to a specific query, the model can focus its learning on high-signal moments. While the current model relies on a pre-trained Region Proposal Network (RPN), which currently caps performance, future work could incorporate spatial constraints—such as tracking the object's movement path—to further refine the grounding beyond feature-level embeddings.

Summary updated

Abstract

We invest the problem of weakly-supervised video grounding, where only video-level sentences are provided. This is a challenging task, and previous Multi-Instance Learning (MIL) based image grounding methods turn to fail in the video domain. Recent work attempts to decompose the video-level MIL into frame-level MIL by applying weighted sentence-frame ranking loss over frames, but it is not robust and does not exploit the rich temporal information in videos. In this work, we address these issues by extending frame-level MIL with a false positive frame-bag constraint and modeling the visual feature consistency in the video. In specific, we design a contextual similarity between semantic and visual features to deal with sparse objects association across frames. Furthermore, we leverage temporal coherence by strengthening the clustering effect of similar features in the visual space. We conduct an extensive evaluation on YouCookII and RoboWatch datasets, and demonstrate our method significantly outperforms prior state-of-the-art methods.
BibTeX
@inproceedings{shi2019not,
  author = {Jing Shi and Jia Xu and Boqing Gong and Chenliang Xu},
  title = {{Not All Frames Are Equal: Weakly-Supervised Video Grounding With Contextual Similarity and Visual Clustering Losses}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  url = {https://openreview.net/forum?id=LIgjGg3G1d},
}
Figure