Divide and Conquer for Single-frame Temporal Action Localization

Chen Ju, Peisen Zhao, Siheng Chen, Ya Zhang, Yanfeng Wang, Qi Tian

Abstract

Single-frame Temporal Action Localization (STAL) is a cost-effective alternative to fully-supervised methods, requiring only a single timestamp per action instance. However, current one-stage approaches struggle with a "coupling issue," where the same thresholding mechanism is used for both counting action instances and determining their temporal boundaries. This paper introduces a novel two-stage framework that decouples these tasks. The "Divide" stage identifies seedframes to isolate action instances into individual clips, while the "Conquer" stage performs precise localization using a proposal-based representation. This approach achieves state-of-the-art performance on THUMOS14, GTEA, and BEOID benchmarks.

Introduction: The Coupling Issue

Temporal Action Localization (TAL) requires identifying the start, end, and category of actions in untrimmed videos. While fully-supervised methods are accurate, they are labor-intensive. Weakly-supervised methods using only video-level labels are cheaper but less precise. Single-frame Temporal Action Localization (STAL) offers a middle ground, using one annotated "seedframe" per action.

Existing STAL methods typically follow a one-stage framework. They estimate frame-level action probabilities and apply a global threshold to identify boundaries. This leads to a "coupling issue": a threshold optimized for counting the number of actions often fails to accurately define their start and end points, and vice versa.

Comparison between the traditional one-stage framework (A) and the proposed two-stage "Divide and Conquer" framework (B).

Methodology: Divide and Conquer

To overcome the coupling issue, the authors propose a two-stage "Divide and Conquer" framework that separates instance counting from location estimation.

1. Divide: The Instance Counting Stage

In this stage, a seedframe detector is trained to identify the annotated timestamps. These detected seedframes serve as anchors to split the long, untrimmed video into multiple smaller clips. The primary goal of this stage is to ensure that each resulting clip contains exactly one complete action instance, effectively solving the counting problem before localization begins.

2. Conquer: The Location Estimation Stage

Once instances are isolated, the model focuses on precise localization within each clip. Unlike frame-based methods that predict probabilities for every frame, this stage uses a proposal-based representation. The model predicts two specific values: the center of the action and its length. This approach inherently enforces temporal smoothness and reduces the search space compared to frame-by-frame classification.

The detailed architecture of the two-stage framework, illustrating the training and inference pipelines for both the Divide and Conquer stages.

Differentiable Mask Generation

A key challenge in proposal-based localization is that action boundaries are discrete and non-differentiable. To enable end-to-end training, the authors developed a gate-approximation network. This module transforms the predicted center and length into a "soft" temporal mask, allowing the system to be trained using standard backpropagation against category labels.

Experimental Results

The framework was evaluated on three major benchmarks: THUMOS14, GTEA, and BEOID. The results demonstrate a significant improvement over previous state-of-the-art methods:

  • THUMOS14: mAP improved by 3.5%.
  • GTEA: mAP improved by 2.7%.
  • BEOID: mAP improved by 4.8%.

Notably, at lower Intersection over Union (IoU) thresholds, this single-frame supervised method outperformed several older fully-supervised models. This suggests that a single, well-placed annotation per action, when processed through a decoupled framework, provides extremely high-quality supervision.

Conclusion

The "Divide and Conquer" strategy effectively addresses the coupling issue in STAL by separating instance counting from boundary estimation. By moving from frame-based to proposal-based representations and utilizing a differentiable mask generator, the model achieves superior localization accuracy. Future research may focus on further refining center estimation and exploring even sparser supervision scenarios.

Summary updated

Abstract

Single-frame temporal action localization (STAL) aims to localize actions in untrimmed videos with only one timestamp annotation for each action instance. Existing methods adopt the one-stage framework but couple the counting goal and the localization goal. This paper proposes a novel two-stage framework for the STAL task with the spirit of divide and conquer. The instance counting stage leverages the location supervision to determine the number of action instances and divide a whole video into multiple video clips, so that each video clip contains only one complete action instance; and the location estimation stage leverages the category supervision to localize the action instance in each video clip. To efficiently represent the action instance in each video clip, we introduce the proposal-based representation, and design a novel differentiable mask generator to enable the end-to-end training supervised by category labels. On THUMOS14, GTEA, and BEOID datasets, our method outperforms state-of-the-art methods by 3.5%, 2.7%, 4.8% mAP on average. And extensive experiments verify the effectiveness of our method.
BibTeX
@inproceedings{ju2021divide,
  author = {Chen Ju and Peisen Zhao and Siheng Chen and Ya Zhang and Yanfeng Wang and Qi Tian},
  title = {{Divide and Conquer for Single-frame Temporal Action Localization}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01320},
}
Figure