ProTéGé: Untrimmed Pretraining for Video Temporal Grounding by Video Temporal Grounding

Lan Wang, Gaurav Mittal, Sandra Sajeev, Ye Yu, Matthew Hall, Vishnu Naresh Boddeti, Mei ChenCVPR 2023 · OpenReview

Abstract: Video temporal grounding (VTG) involves localizing a natural language query within an arbitrarily long, untrimmed video. Despite the long-form nature of the task, existing VTG methods typically rely on video backbones pretrained on short, trimmed clips. This creates a "temporal boundary gap" where models struggle to distinguish precise start and end points. ProTéGé addresses this by performing VTG-based pretraining on untrimmed videos. By repurposing the HowTo100M dataset and introducing a noise-robust Video-Text Similarity-based Grounding Module (VT-SGM), ProTéGé learns features that are inherently sensitive to temporal dynamics. Experiments demonstrate that ProTéGé significantly outperforms traditional trimmed-video backbones across supervised, weakly supervised, and zero-shot settings.

The Problem: The "Trimmed" Blind Spot

Video Temporal Grounding (VTG) requires an AI to pinpoint the exact start and end times of an action described in text. However, the "backbones" used to extract video features are almost exclusively trained on datasets like Kinetics, which consist of short, 10-second "trimmed" clips.

When a model only sees isolated snippets, it never learns the context of what happens before or after an action. This leads to a lack of temporal sensitivity: the model might recognize "pouring water" but cannot distinguish the exact frame where the pouring begins from the frame where the person is just holding the pitcher. This disconnect makes video-text alignment less distinguishable between correct and incorrect locations.

Comparison of trimmed vs. untrimmed pretraining. Trimmed pretraining leads to less distinguishable video-text alignment at boundaries, whereas ProTéGé (untrimmed) aligns more closely with ground truth.

The ProTéGé Framework: Bridging the Gap

To solve this, the authors developed ProTéGé (Untrimmed Pretraining for Video Temporal Gounding). The framework introduces three main innovations to enable effective pretraining on long-form video:

1. Synthesizing a VTG Dataset

The team utilized the HowTo100M dataset, which contains over a million narrated YouTube tutorials. Since these lack explicit temporal grounding labels, they aggregated consecutive subtitles to create complex text queries that span multiple segments, mimicking the structure of downstream VTG tasks.

2. Video-Text Similarity-based Grounding Module (VT-SGM)

Narrated videos are notoriously "noisy"—what a person says doesn't always perfectly align with what they are doing at that exact second. To handle this, ProTéGé uses a 2D proposal grid that calculates simple cosine similarities between text and video segments. This approach is more robust to alignment noise than complex fusion architectures, preventing the model from overfitting to imperfect data.

3. A Triple-Loss Objective

The model is trained using three distinct losses to ensure both global and local alignment:

  • Localization Loss ($\\mathcal{L}_{loc}$): Encourages the model to predict the correct temporal window using a combination of cross-entropy and a "soft" Manhattan distance penalty.
  • Inter-Video Alignment: A contrastive loss that ensures a text query matches its corresponding video better than other videos in a batch.
  • Intra-Video Alignment: Forces the model to distinguish the ground-truth segment from other, less relevant parts of the same video.
The ProTéGé architecture: (a) Dataset synthesis from untrimmed video, (b) The VT-SGM module, and (c-e) The triple-loss objective for robust pretraining.

Key Results and Impact

ProTéGé's untrimmed pretraining yields features that are significantly more effective for localization tasks than standard backbones:

  • Supervised Learning: On the Charades-STA dataset, ProTéGé improved the [email protected] metric (recall at 0.5 IoU) by over 7% compared to a standard Swin-B backbone.
  • Weakly Supervised & Zero-Shot: The model showed remarkable transferability. In zero-shot scenarios on ActivityNet-Captions, where the model had never seen the specific downstream data, it outperformed existing methods by over 2% mIoU.
  • Efficiency: By using a simpler similarity-based module, the framework remains computationally efficient while being more robust to the inherent noise of large-scale web video datasets.

Conclusion

ProTéGé demonstrates that the "trimmed video" paradigm is a bottleneck for long-form video understanding. By pretraining on untrimmed videos with a noise-robust grounding objective, AI can develop a much finer sense of time and boundaries. While challenges remain regarding the temporal offset between speech and action in narrated videos, ProTéGé sets a new standard for how we prepare models for complex, real-world video tasks.

Summary updated

Abstract

Video temporal grounding (VTG) is the task of localizing a given natural language text query in an arbitrarily long untrimmed video. While the task involves untrimmed videos, all existing VTG methods leverage features from video backbones pretrained on trimmed videos. This is largely due to the lack of large-scale well-annotated VTG dataset to perform pretraining. As a result, the pretrained features lack a notion of temporal boundaries leading to the video-text alignment being less distinguishable between correct and incorrect locations. We present ProTéGé as the first method to perform VTG-based untrimmed pretraining to bridge the gap between trimmed pretrained backbones and downstream VTG tasks. ProTéGé reconfigures the HowTo100M dataset, with noisily correlated video-text pairs, into a VTG dataset and introduces a novel Video-Text Similarity-based Grounding Module and a pretraining objective to make pretraining robust to noise in HowTo100M. Extensive experiments on multiple datasets across downstream tasks with all variations of supervision validate that pretrained features from ProTéGé can significantly outperform features from trimmed pretrained backbones on VTG.
BibTeX
@inproceedings{wang2023protege,
  author = {Lan Wang and Gaurav Mittal and Sandra Sajeev and Ye Yu and Matthew Hall and Vishnu Naresh Boddeti and Mei Chen},
  title = {{ProTéGé: Untrimmed Pretraining for Video Temporal Grounding by Video Temporal Grounding}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=Ke6hLTjff1},
}
Figure