Abstract
Temporal sentence grounding (TSG) aims to localize specific video segments based on natural language queries. While supervised methods achieve high accuracy, they rely on expensive timestamp annotations. Weakly supervised approaches, typically based on Multiple Instance Learning (MIL), often suffer from noisy labels and a lack of precise positive guidance. This paper introduces a novel uncertainty-guided self-training framework that leverages a Bayesian teacher network to estimate prediction uncertainty and a temporal cycle-consistency mechanism. By filtering unreliable pseudo-labels and enforcing temporal robustness, the proposed method significantly enhances the performance of various backbone architectures, achieving state-of-the-art results on major benchmarks.
Introduction
Temporal sentence grounding is a fundamental task in video understanding, often described as the "needle in a haystack" problem. Given a sentence query like "The person puts the book on the shelf," a model must identify the exact start and end times of that action within a longer video sequence.
While fully supervised models have made significant strides, the requirement for massive amounts of manually annotated timestamps is a major bottleneck. Weakly supervised methods attempt to learn these associations using only video-level labels—knowing a sentence occurs somewhere in the video without knowing exactly when. However, standard approaches like Multiple Instance Learning (MIL) often fail to distinguish the true ground truth from other vaguely relevant segments. This research addresses this gap by providing a reliable "positive" guidance signal through a self-training framework.
Uncertainty-Guided Self-training Framework
The core of the proposed approach is a teacher-student architecture designed to refine localization boundaries through mutual learning and uncertainty estimation.
Bayesian Teacher and Uncertainty Estimation
Because the teacher network is also trained under weak supervision, its predictions (pseudo-labels) can be noisy. To prevent the student from learning these errors, the authors employ a Bayesian Teacher using Monte Carlo Dropout. By performing multiple forward passes with different dropout masks, the model calculates the variance (uncertainty) of its outputs. This uncertainty acts as a weight: the student model only learns from the teacher's predictions when the teacher is highly confident, effectively filtering out noise.
Temporal Cycle Consistency and Augmentation
To improve robustness, the framework utilizes a dual-augmentation strategy. The teacher receives weakly augmented data, while the student is challenged with strong temporal augmentations, such as scaling and shifting. A Temporal Cycle Consistency mechanism ensures that if a video is transformed, the predicted segments transform accordingly. This forces the model to understand the relative timing and boundaries of actions rather than relying on static visual cues.
Masked Reconstruction
In addition to temporal localization, the model is tasked with a reconstruction objective. By masking parts of the text query and requiring the model to reconstruct them using only the selected video segment, the framework ensures that the chosen segment contains the necessary semantic information to match the query.
Experimental Evaluation
The framework was evaluated on two gold-standard benchmarks:
- Charades-STA: A dataset featuring 12,408 training pairs with relatively short, simple sentences.
- ActivityNet Captions: A larger, more complex dataset (~37k training pairs) involving longer videos and descriptive, multi-action sentences.
For complex queries in ActivityNet, the authors utilized Semantic Role Labeling (SRL) to decompose sentences into core components, helping the model focus on essential actions.
Results and Discussion
The proposed method achieved state-of-the-art (SOTA) results across both datasets. On Charades-STA, it reached 45.20 mIoU, and on ActivityNet Captions, it achieved 41.02 mIoU.
A key strength of this framework is its "plug-and-play" nature. As shown in the experimental results, the self-training process consistently improved the performance of three distinct backbone architectures: CPL, CNM, and WSTAN. This demonstrates that the uncertainty-guided approach is a versatile refiner that can be applied to various underlying models to boost their localization accuracy.
Conclusion
This paper presents a robust self-training framework for weakly supervised temporal sentence grounding. By integrating Bayesian uncertainty estimation with temporal cycle consistency, the model successfully filters noisy supervision and learns more precise action boundaries. While the method's effectiveness is currently tied to the quality of the initial backbone, it represents a significant step toward high-performance video grounding without the need for exhaustive manual annotation. Future work may explore improving the diversity of proposals to further enhance top-k retrieval performance.
Summary updated