Weakly Supervised Temporal Sentence Grounding with Uncertainty-Guided Self-training

Yifei Huang, Lijin Yang, Yoichi Sato

Abstract

Temporal sentence grounding (TSG) aims to localize specific video segments based on natural language queries. While supervised methods achieve high accuracy, they rely on expensive timestamp annotations. Weakly supervised approaches, typically based on Multiple Instance Learning (MIL), often suffer from noisy labels and a lack of precise positive guidance. This paper introduces a novel uncertainty-guided self-training framework that leverages a Bayesian teacher network to estimate prediction uncertainty and a temporal cycle-consistency mechanism. By filtering unreliable pseudo-labels and enforcing temporal robustness, the proposed method significantly enhances the performance of various backbone architectures, achieving state-of-the-art results on major benchmarks.

Introduction

Temporal sentence grounding is a fundamental task in video understanding, often described as the "needle in a haystack" problem. Given a sentence query like "The person puts the book on the shelf," a model must identify the exact start and end times of that action within a longer video sequence.

While fully supervised models have made significant strides, the requirement for massive amounts of manually annotated timestamps is a major bottleneck. Weakly supervised methods attempt to learn these associations using only video-level labels—knowing a sentence occurs somewhere in the video without knowing exactly when. However, standard approaches like Multiple Instance Learning (MIL) often fail to distinguish the true ground truth from other vaguely relevant segments. This research addresses this gap by providing a reliable "positive" guidance signal through a self-training framework.

Uncertainty-Guided Self-training Framework

The core of the proposed approach is a teacher-student architecture designed to refine localization boundaries through mutual learning and uncertainty estimation.

Overview of the proposed uncertainty-guided self-training framework, illustrating the teacher-student cycle consistency and the Bayesian teacher's role in uncertainty estimation.

Bayesian Teacher and Uncertainty Estimation

Because the teacher network is also trained under weak supervision, its predictions (pseudo-labels) can be noisy. To prevent the student from learning these errors, the authors employ a Bayesian Teacher using Monte Carlo Dropout. By performing multiple forward passes with different dropout masks, the model calculates the variance (uncertainty) of its outputs. This uncertainty acts as a weight: the student model only learns from the teacher's predictions when the teacher is highly confident, effectively filtering out noise.

Temporal Cycle Consistency and Augmentation

To improve robustness, the framework utilizes a dual-augmentation strategy. The teacher receives weakly augmented data, while the student is challenged with strong temporal augmentations, such as scaling and shifting. A Temporal Cycle Consistency mechanism ensures that if a video is transformed, the predicted segments transform accordingly. This forces the model to understand the relative timing and boundaries of actions rather than relying on static visual cues.

Masked Reconstruction

In addition to temporal localization, the model is tasked with a reconstruction objective. By masking parts of the text query and requiring the model to reconstruct them using only the selected video segment, the framework ensures that the chosen segment contains the necessary semantic information to match the query.

Experimental Evaluation

The framework was evaluated on two gold-standard benchmarks:

  • Charades-STA: A dataset featuring 12,408 training pairs with relatively short, simple sentences.
  • ActivityNet Captions: A larger, more complex dataset (~37k training pairs) involving longer videos and descriptive, multi-action sentences.

For complex queries in ActivityNet, the authors utilized Semantic Role Labeling (SRL) to decompose sentences into core components, helping the model focus on essential actions.

Results and Discussion

The proposed method achieved state-of-the-art (SOTA) results across both datasets. On Charades-STA, it reached 45.20 mIoU, and on ActivityNet Captions, it achieved 41.02 mIoU.

Comparative performance ([email protected]) of various backbone models with and without the proposed self-training framework on ActivityNet Captions and Charades-STA.

A key strength of this framework is its "plug-and-play" nature. As shown in the experimental results, the self-training process consistently improved the performance of three distinct backbone architectures: CPL, CNM, and WSTAN. This demonstrates that the uncertainty-guided approach is a versatile refiner that can be applied to various underlying models to boost their localization accuracy.

Conclusion

This paper presents a robust self-training framework for weakly supervised temporal sentence grounding. By integrating Bayesian uncertainty estimation with temporal cycle consistency, the model successfully filters noisy supervision and learns more precise action boundaries. While the method's effectiveness is currently tied to the quality of the initial backbone, it represents a significant step toward high-performance video grounding without the need for exhaustive manual annotation. Future work may explore improving the diversity of proposals to further enhance top-k retrieval performance.

Summary updated

Abstract

The task of weakly supervised temporal sentence grounding aims at finding the corresponding temporal moments of a language description in the video, given video-language correspondence only at video-level. Most existing works select mismatched video-language pairs as negative samples and train the model to generate better positive proposals that are distinct from the negative ones. However, due to the complex temporal structure of videos, proposals distinct from the negative ones may correspond to several video segments but not necessarily the correct ground truth. To alleviate this problem, we propose an uncertainty-guided self-training technique to provide extra self-supervision signal to guide the weakly-supervised learning. The self-training process is based on teacher-student mutual learning with weak-strong augmentation, which enables the teacher network to generate relatively more reliable outputs compared to the student network, so that the student network can learn from the teacher's output. Since directly applying existing self-training methods in this task easily causes error accumulation, we specifically design two techniques in our selftraining method: (1) we construct a Bayesian teacher network, leveraging its uncertainty as a weight to suppress the noisy teacher supervisory signals; (2) we leverage the cycle consistency brought by temporal data augmentation to perform mutual learning between the two networks. Experiments demonstrate our method's superiority on Charades-STA and ActivityNet Captions datasets. We also show in the experiment that our self-training method can be applied to improve the performance of multiple backbone methods.
BibTeX
@inproceedings{huang2023weakly,
  author = {Yifei Huang and Lijin Yang and Yoichi Sato},
  title = {{Weakly Supervised Temporal Sentence Grounding with Uncertainty-Guided Self-training}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01813},
}
Figure