Temporal Structure Mining for Weakly Supervised Action Detection

Tan Yu, Zhou Ren, Yuncheng Li, Enxu Yan, Ning Xu, Junsong YuanICCV 2019 · DOI

Abstract

Weakly Supervised Action Detection (WSAD) aims to localize actions in untrimmed videos using only video-level category labels. Traditional methods often treat video segments as independent snapshots, ignoring the inherent temporal structure of actions. This paper introduces Temporal Structure Mining (TSM), which models actions as a sequence of evolving phases. By incorporating a circulant phase evolution and a recurrent path discovery algorithm, TSM effectively captures the latent temporal logic of actions, significantly improving localization accuracy on benchmark datasets like THUMOS14 and ActivityNet.

Introduction

Action detection—identifying both the "what" and "when" of activities in video—is a fundamental task in computer vision. While fully supervised methods have achieved impressive results, they rely on precise frame-level annotations, which are prohibitively expensive to collect at scale. Weakly Supervised Action Detection (WSAD) offers a more scalable alternative by training models using only video-level labels (e.g., "this video contains a high jump").

However, most existing WSAD approaches suffer from a lack of temporal context. They typically identify high-scoring segments independently, failing to distinguish between the actual action and similar-looking background noise or "pre-action" movements. To address this, we propose modeling actions as structured processes with distinct phases.

Temporal Structure Mining (TSM)

The core philosophy of TSM is that an action is not a static state but a process with a beginning, middle, and end.

Phase Modeling and Evolution

Instead of a single classifier for an action category, TSM employs multiple "phase filters" ($a_1, a_2, \dots, a_M$) to represent different stages of an activity. To handle untrimmed videos containing multiple action instances, we introduce a background phase ($a_0$) and enforce a circulant phase evolution.

The TSM phase evolution model incorporates a background phase (a0) and allows for multiple action instances through a circulant transition.

Unlike previous linear models that could only handle a single action per clip, TSM's circulant structure allows the model to cycle from the final action phase back to the background, enabling the detection of multiple instances in a single long video.

Framework Overview and Path Discovery

The TSM framework processes segment features through phase filters to generate a scores table. Since the exact phase of each segment is unknown during training, we treat them as hidden variables.

Overview of the TSM framework: Segment features are passed through phase filters to create a scores table, followed by circulant path discovery.

We use Dynamic Programming to find the "maximal circulant path"—the sequence of phases that maximizes the total score while adhering to the circulant order. To improve training stability and prevent the model from overfitting to noisy segments, we implement a Soft-max Path Discovery variant. This approach considers a weighted ensemble of high-scoring paths rather than just the single best one, providing a richer signal for optimization.

Experimental Results

We evaluated TSM on three major benchmarks: THUMOS14, ActivityNet 1.2, and ActivityNet 1.3. In all experiments, the model was trained using only video-level category labels.

Performance Comparison

TSM significantly outperforms existing state-of-the-art weakly supervised methods. On THUMOS14, TSM achieved a mean Average Precision (mAP) of 24.5% at an Intersection over Union (IoU) threshold of 0.5, a substantial improvement over STPN (16.9%) and Auto-Loc (21.2%). Similar gains were observed on ActivityNet 1.3, where TSM reached an mAP of 19.0% at IoU=0.75.

Key Insights

  • Number of Phases: Our analysis showed that using 3 phases provides the best balance between capturing structure and avoiding overfitting. Models using only a single phase performed significantly worse.
  • Circulant vs. Linear: The circulant structure was crucial for performance on untrimmed videos, as it naturally handles the transition between multiple action instances and background noise.

Conclusion

Temporal Structure Mining (TSM) demonstrates that even under weak supervision, modeling the internal temporal logic of actions is highly beneficial. By treating actions as multi-phase processes and utilizing a circulant path discovery mechanism, TSM bridges the gap between video-level labels and frame-level localization. Future work will explore adaptive phase counts for different action types and more complex background modeling to further refine detection in cluttered environments.

Summary updated

BibTeX
@inproceedings{yu2019temporal,
  author = {Tan Yu and Zhou Ren and Yuncheng Li and Enxu Yan and Ning Xu and Junsong Yuan},
  title = {{Temporal Structure Mining for Weakly Supervised Action Detection}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00562},
}
Figure