Weakly Supervised Temporal Action Localization Through Contrast Based Evaluation Networks

Ziyi Liu, Le Wang, Qilin Zhang, Zhanning Gao, Zhenxing Niu, Nanning Zheng, Gang Hua

Abstract

Weakly Supervised Temporal Action Localization (WS-TAL) is a challenging task that involves identifying the temporal boundaries of actions in untrimmed videos using only video-level categorical labels. Traditional WS-TAL methods often rely on snippet-level classification scores, ignoring the temporal context necessary for precise boundary detection. We present CleanNet (Contrast-based Localization EvaluAtioN Network), a framework presented at ICCV 2019 that leverages "temporal contrast" to evaluate action proposals. By modeling the "edgeness" of actions—the sharp changes in classification probability at the start and end of activities—CleanNet effectively localizes actions without frame-level supervision. Our method incorporates a sparse temporal sampling strategy to align the receptive field with varying action lengths and employs a joint training scheme to filter background noise. CleanNet achieves state-of-the-art results on THUMOS14 and ActivityNet v1.2, demonstrating performance comparable to early fully supervised methods.

Introduction

Temporal Action Localization (TAL) is critical for video understanding, yet the high cost of frame-level boundary annotations limits the scalability of fully supervised approaches. Weakly Supervised TAL (WS-TAL) offers a more scalable alternative by utilizing only video-level labels. However, existing WS-TAL techniques often struggle with boundary precision because they treat video snippets independently. CleanNet addresses this by explicitly modeling the temporal relationships between snippets, focusing on the contrast between an action and its immediate background.

Methodology

The CleanNet architecture is composed of three integrated modules: Feature Embedding, Action Classification, and Action Localization.

The CleanNet framework: Features are embedded and passed to classification and localization modules.

The core innovation lies in the Action Proposal Evaluator. Instead of relying solely on high classification scores, CleanNet calculates a "contrast score" that measures the "edgeness" of a proposal. It compares the classification predictions within a proposed segment to the snippets immediately preceding and following it. This allows the model to identify "rising edges" (starts) and "falling edges" (ends) of activities.

Detailed view of the Action Proposal Evaluator modeling temporal edgeness.

To handle varying action durations, the Action Proposal Generator utilizes a sampling strategy inspired by Temporal Segment Networks (TSN). This ensures the model's receptive field is appropriately sized for the temporal anchor being evaluated. Furthermore, the classification and localization modules are trained jointly in a mutualistic fashion; the localization module identifies background snippets, which are then excluded from classification training to improve the signal-to-noise ratio.

Experiments and Results

CleanNet was evaluated on the THUMOS14 and ActivityNet v1.2 benchmarks. On THUMOS14, CleanNet achieved a mean Average Precision (mAP) of 23.9% at an IoU threshold of 0.5, significantly outperforming the 15.4% baseline (UntrimmedNet) and contemporary models like AutoLoc (21.2%). On ActivityNet v1.2, it reached an average mAP of 21.6%, surpassing W-TALC (18.0%) and AutoLoc (16.0%). Notably, CleanNet's performance on THUMOS14 is competitive with several early fully supervised methods, such as S-CNN (19.0%) and CDC (23.3%), despite the lack of expensive boundary-level supervision.

Conclusion

CleanNet demonstrates that temporal contrast is a powerful cue for action localization in the absence of dense annotations. By focusing on the transitions between action and background rather than just independent snippet scores, the model achieves superior boundary precision. While precise localization at high IoU thresholds remains a challenge for weakly supervised methods, CleanNet's results suggest that modeling temporal relationships is essential for bridging the gap with fully supervised systems.

Summary updated

Abstract

Weakly-supervised temporal action localization (WS-TAL) is a promising but challenging task with only video-level action categorical labels available during training. Without requiring temporal action boundary annotations in training data, WS-TAL could possibly exploit automatically retrieved video tags as video-level labels. However, such coarse video-level supervision inevitably incurs confusions, especially in untrimmed videos containing multiple action instances. To address this challenge, we propose the Contrast-based Localization EvaluAtioN Network (CleanNet) with our new action proposal evaluator, which provides pseudo-supervision by leveraging the temporal contrast in snippet-level action classification predictions. Essentially, the new action proposal evaluator enforces an additional temporal contrast constraint so that high-evaluation-score action proposals are more likely to coincide with true action instances. Moreover, the new action localization module is an integral part of CleanNet which enables end-to-end training. This is in contrast to many existing WS-TAL methods where action localization is merely a post-processing step. Experiments on THUMOS14 and ActivityNet datasets validate the efficacy of CleanNet against existing state-ofthe- art WS-TAL algorithms.
BibTeX
@inproceedings{liu2019weakly,
  author = {Ziyi Liu and Le Wang and Qilin Zhang and Zhanning Gao and Zhenxing Niu and Nanning Zheng and Gang Hua},
  title = {{Weakly Supervised Temporal Action Localization Through Contrast Based Evaluation Networks}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00400},
}
Figure