Abstract: Video temporal grounding (VTG) involves localizing a natural language query within an arbitrarily long, untrimmed video. Despite the long-form nature of the task, existing VTG methods typically rely on video backbones pretrained on short, trimmed clips. This creates a "temporal boundary gap" where models struggle to distinguish precise start and end points. ProTéGé addresses this by performing VTG-based pretraining on untrimmed videos. By repurposing the HowTo100M dataset and introducing a noise-robust Video-Text Similarity-based Grounding Module (VT-SGM), ProTéGé learns features that are inherently sensitive to temporal dynamics. Experiments demonstrate that ProTéGé significantly outperforms traditional trimmed-video backbones across supervised, weakly supervised, and zero-shot settings.
The Problem: The "Trimmed" Blind Spot
Video Temporal Grounding (VTG) requires an AI to pinpoint the exact start and end times of an action described in text. However, the "backbones" used to extract video features are almost exclusively trained on datasets like Kinetics, which consist of short, 10-second "trimmed" clips.
When a model only sees isolated snippets, it never learns the context of what happens before or after an action. This leads to a lack of temporal sensitivity: the model might recognize "pouring water" but cannot distinguish the exact frame where the pouring begins from the frame where the person is just holding the pitcher. This disconnect makes video-text alignment less distinguishable between correct and incorrect locations.
The ProTéGé Framework: Bridging the Gap
To solve this, the authors developed ProTéGé (Untrimmed Pretraining for Video Temporal Gounding). The framework introduces three main innovations to enable effective pretraining on long-form video:
1. Synthesizing a VTG Dataset
The team utilized the HowTo100M dataset, which contains over a million narrated YouTube tutorials. Since these lack explicit temporal grounding labels, they aggregated consecutive subtitles to create complex text queries that span multiple segments, mimicking the structure of downstream VTG tasks.
2. Video-Text Similarity-based Grounding Module (VT-SGM)
Narrated videos are notoriously "noisy"—what a person says doesn't always perfectly align with what they are doing at that exact second. To handle this, ProTéGé uses a 2D proposal grid that calculates simple cosine similarities between text and video segments. This approach is more robust to alignment noise than complex fusion architectures, preventing the model from overfitting to imperfect data.
3. A Triple-Loss Objective
The model is trained using three distinct losses to ensure both global and local alignment:
- Localization Loss ($\\mathcal{L}_{loc}$): Encourages the model to predict the correct temporal window using a combination of cross-entropy and a "soft" Manhattan distance penalty.
- Inter-Video Alignment: A contrastive loss that ensures a text query matches its corresponding video better than other videos in a batch.
- Intra-Video Alignment: Forces the model to distinguish the ground-truth segment from other, less relevant parts of the same video.
Key Results and Impact
ProTéGé's untrimmed pretraining yields features that are significantly more effective for localization tasks than standard backbones:
- Supervised Learning: On the Charades-STA dataset, ProTéGé improved the [email protected] metric (recall at 0.5 IoU) by over 7% compared to a standard Swin-B backbone.
- Weakly Supervised & Zero-Shot: The model showed remarkable transferability. In zero-shot scenarios on ActivityNet-Captions, where the model had never seen the specific downstream data, it outperformed existing methods by over 2% mIoU.
- Efficiency: By using a simpler similarity-based module, the framework remains computationally efficient while being more robust to the inherent noise of large-scale web video datasets.
Conclusion
ProTéGé demonstrates that the "trimmed video" paradigm is a bottleneck for long-form video understanding. By pretraining on untrimmed videos with a noise-robust grounding objective, AI can develop a much finer sense of time and boundaries. While challenges remain regarding the temporal offset between speech and action in narrated videos, ProTéGé sets a new standard for how we prepare models for complex, real-world video tasks.
Summary updated