Abstract
Weakly Supervised Temporal Action Localization (WS-TAL) is a challenging task that involves identifying the temporal boundaries of actions in untrimmed videos using only video-level categorical labels. Traditional WS-TAL methods often rely on snippet-level classification scores, ignoring the temporal context necessary for precise boundary detection. We present CleanNet (Contrast-based Localization EvaluAtioN Network), a framework presented at ICCV 2019 that leverages "temporal contrast" to evaluate action proposals. By modeling the "edgeness" of actions—the sharp changes in classification probability at the start and end of activities—CleanNet effectively localizes actions without frame-level supervision. Our method incorporates a sparse temporal sampling strategy to align the receptive field with varying action lengths and employs a joint training scheme to filter background noise. CleanNet achieves state-of-the-art results on THUMOS14 and ActivityNet v1.2, demonstrating performance comparable to early fully supervised methods.
Introduction
Temporal Action Localization (TAL) is critical for video understanding, yet the high cost of frame-level boundary annotations limits the scalability of fully supervised approaches. Weakly Supervised TAL (WS-TAL) offers a more scalable alternative by utilizing only video-level labels. However, existing WS-TAL techniques often struggle with boundary precision because they treat video snippets independently. CleanNet addresses this by explicitly modeling the temporal relationships between snippets, focusing on the contrast between an action and its immediate background.
Methodology
The CleanNet architecture is composed of three integrated modules: Feature Embedding, Action Classification, and Action Localization.
The core innovation lies in the Action Proposal Evaluator. Instead of relying solely on high classification scores, CleanNet calculates a "contrast score" that measures the "edgeness" of a proposal. It compares the classification predictions within a proposed segment to the snippets immediately preceding and following it. This allows the model to identify "rising edges" (starts) and "falling edges" (ends) of activities.
To handle varying action durations, the Action Proposal Generator utilizes a sampling strategy inspired by Temporal Segment Networks (TSN). This ensures the model's receptive field is appropriately sized for the temporal anchor being evaluated. Furthermore, the classification and localization modules are trained jointly in a mutualistic fashion; the localization module identifies background snippets, which are then excluded from classification training to improve the signal-to-noise ratio.
Experiments and Results
CleanNet was evaluated on the THUMOS14 and ActivityNet v1.2 benchmarks. On THUMOS14, CleanNet achieved a mean Average Precision (mAP) of 23.9% at an IoU threshold of 0.5, significantly outperforming the 15.4% baseline (UntrimmedNet) and contemporary models like AutoLoc (21.2%). On ActivityNet v1.2, it reached an average mAP of 21.6%, surpassing W-TALC (18.0%) and AutoLoc (16.0%). Notably, CleanNet's performance on THUMOS14 is competitive with several early fully supervised methods, such as S-CNN (19.0%) and CDC (23.3%), despite the lack of expensive boundary-level supervision.
Conclusion
CleanNet demonstrates that temporal contrast is a powerful cue for action localization in the absence of dense annotations. By focusing on the transitions between action and background rather than just independent snippet scores, the model achieves superior boundary precision. While precise localization at high IoU thresholds remains a challenge for weakly supervised methods, CleanNet's results suggest that modeling temporal relationships is essential for bridging the gap with fully supervised systems.
Summary updated