Abstract
Single-frame Temporal Action Localization (STAL) is a cost-effective alternative to fully-supervised methods, requiring only a single timestamp per action instance. However, current one-stage approaches struggle with a "coupling issue," where the same thresholding mechanism is used for both counting action instances and determining their temporal boundaries. This paper introduces a novel two-stage framework that decouples these tasks. The "Divide" stage identifies seedframes to isolate action instances into individual clips, while the "Conquer" stage performs precise localization using a proposal-based representation. This approach achieves state-of-the-art performance on THUMOS14, GTEA, and BEOID benchmarks.
Introduction: The Coupling Issue
Temporal Action Localization (TAL) requires identifying the start, end, and category of actions in untrimmed videos. While fully-supervised methods are accurate, they are labor-intensive. Weakly-supervised methods using only video-level labels are cheaper but less precise. Single-frame Temporal Action Localization (STAL) offers a middle ground, using one annotated "seedframe" per action.
Existing STAL methods typically follow a one-stage framework. They estimate frame-level action probabilities and apply a global threshold to identify boundaries. This leads to a "coupling issue": a threshold optimized for counting the number of actions often fails to accurately define their start and end points, and vice versa.
Methodology: Divide and Conquer
To overcome the coupling issue, the authors propose a two-stage "Divide and Conquer" framework that separates instance counting from location estimation.
1. Divide: The Instance Counting Stage
In this stage, a seedframe detector is trained to identify the annotated timestamps. These detected seedframes serve as anchors to split the long, untrimmed video into multiple smaller clips. The primary goal of this stage is to ensure that each resulting clip contains exactly one complete action instance, effectively solving the counting problem before localization begins.
2. Conquer: The Location Estimation Stage
Once instances are isolated, the model focuses on precise localization within each clip. Unlike frame-based methods that predict probabilities for every frame, this stage uses a proposal-based representation. The model predicts two specific values: the center of the action and its length. This approach inherently enforces temporal smoothness and reduces the search space compared to frame-by-frame classification.
Differentiable Mask Generation
A key challenge in proposal-based localization is that action boundaries are discrete and non-differentiable. To enable end-to-end training, the authors developed a gate-approximation network. This module transforms the predicted center and length into a "soft" temporal mask, allowing the system to be trained using standard backpropagation against category labels.
Experimental Results
The framework was evaluated on three major benchmarks: THUMOS14, GTEA, and BEOID. The results demonstrate a significant improvement over previous state-of-the-art methods:
- THUMOS14: mAP improved by 3.5%.
- GTEA: mAP improved by 2.7%.
- BEOID: mAP improved by 4.8%.
Notably, at lower Intersection over Union (IoU) thresholds, this single-frame supervised method outperformed several older fully-supervised models. This suggests that a single, well-placed annotation per action, when processed through a decoupled framework, provides extremely high-quality supervision.
Conclusion
The "Divide and Conquer" strategy effectively addresses the coupling issue in STAL by separating instance counting from boundary estimation. By moving from frame-based to proposal-based representations and utilizing a differentiable mask generator, the model achieves superior localization accuracy. Future research may focus on further refining center estimation and exploring even sparser supervision scenarios.
Summary updated