TL;DR
This paper introduces a novel two-stage framework for weakly supervised temporal action localization by enforcing consistency across different temporal resolutions. By leveraging the fact that video content remains semantically identical regardless of playback speed, the method generates and iteratively refines high-quality frame-level pseudo-labels, significantly boosting localization accuracy without requiring expensive frame-by-frame annotations.
What Problem Does This Paper Address?
Temporal Action Localization (TAL) aims to pinpoint the exact start and end times of specific actions within long, untrimmed videos. While fully supervised models excel at this, they require meticulous, frame-level boundary annotations, which are incredibly expensive and highly subjective to produce.
Weakly Supervised TAL (WSTAL) offers a cheaper alternative by training models using only video-level labels (e.g., knowing a video contains a "frisbee catch" without knowing when it happens). However, standard WSTAL methods rely on Class Activation Sequences (CAS) that tend to only activate during the most distinct, "discriminative" frames of an action (like the exact moment the frisbee is caught), completely missing the preparation or wind-down phases and leading to fragmented or incomplete localizations.
Methods
The authors tackle this issue by exploiting temporal multi-resolution information in a two-stage approach:
- Initial Label Generation (ILG) Module: Built on top of a two-stream (RGB and optical flow) backbone, this module feeds both original-resolution and temporally scaled (resampled) video features into the network. It introduces two self-supervised constraints to clean up the initial predictions:
- Temporal Multi-Resolution Consistency Loss: Forces the network to output identical action predictions for the same video content, regardless of its temporal scale (speed).
-
Cross-Stream Consistency Loss: Aligns predictions between the appearance (RGB) and motion (optical flow) streams. High-confidence predictions from this module form the initial frame-level pseudo-labels.
-
Progressive Temporal Label Refinement (PTLR) Framework: An iterative co-training framework consisting of two distinct networks that refine each other's pseudo-labels over multiple stages:
- Network-OTS (Original Temporal Scale): Generates predictions from the original video duration.
- Network-RTS (Reduced Temporal Scale): Uses a multi-branch structure with 1D convolutions to extract features at multiple reduced temporal resolutions, which are then upsampled and combined.
Datasets
The approach was validated on two standard video action benchmarks:
- THUMOS14: Features 20 action classes. The authors used 200 validation videos for training (with video-level labels) and 212 test videos for evaluation.
- ActivityNet v1.3: A much larger dataset containing nearly 20,000 videos across 200 activity classes. Models were trained on the training set and evaluated on the validation set.
Key Insights and Ideas
- Temporal Invariance as a Supervisor: The core "aha moment" is that a video's meaning doesn't change if you speed it up or slow it down. Standard networks often yield wildly inconsistent frame-level predictions when a video is temporally scaled; forcing multi-resolution consistency acts as a powerful, annotation-free regularizer.
- Co-Training Across Scales: Network-OTS and Network-RTS act as two different "views" of the same data. Exchanging pseudo-labels between an original-scale network and a multi-scale reduced network helps recover full action boundaries rather than just peak action moments.
- Progressive Threshold Lowering: By gradually lowering the selection threshold across training stages, the model safely transitions from learning only from the easiest, most obvious frames to capturing subtle, less discriminative parts of the action sequence.
Key Results
- THUMOS14: At an Intersection-over-Union (IoU) threshold of 0.5, the proposed method achieves a mean Average Precision (mAP) of 29.7%, outperforming the baseline by 2.7% and beating several older fully supervised methods.
- ActivityNet v1.3: It achieves an average mAP of 23.7%, a 1.5% improvement over the baseline.
- Qualitative Improvements: Visualizations demonstrate that while baseline methods split a single continuous action into fragmented segments because they miss less distinct frames, this multi-resolution approach successfully localizes the entire action duration smoothly.
Limitations and Future Work
The authors note an inherent trade-off in the progressive refinement stages: as more stages are added and the selection threshold is lowered to capture more frames, the precision of the pseudo-labels gradually decreases. This happens because lowering the bar inevitably introduces background noise into the training set. Additionally, the method relies on pre-extracted, frozen feature extractors (I3D), meaning it does not learn end-to-end representations directly from raw video pixels. Mitigating the precision drop in later stages remains an open challenge for future work.
Summary updated