Abstract
Weakly Supervised Action Detection (WSAD) aims to localize actions in untrimmed videos using only video-level category labels. Traditional methods often treat video segments as independent snapshots, ignoring the inherent temporal structure of actions. This paper introduces Temporal Structure Mining (TSM), which models actions as a sequence of evolving phases. By incorporating a circulant phase evolution and a recurrent path discovery algorithm, TSM effectively captures the latent temporal logic of actions, significantly improving localization accuracy on benchmark datasets like THUMOS14 and ActivityNet.
Introduction
Action detection—identifying both the "what" and "when" of activities in video—is a fundamental task in computer vision. While fully supervised methods have achieved impressive results, they rely on precise frame-level annotations, which are prohibitively expensive to collect at scale. Weakly Supervised Action Detection (WSAD) offers a more scalable alternative by training models using only video-level labels (e.g., "this video contains a high jump").
However, most existing WSAD approaches suffer from a lack of temporal context. They typically identify high-scoring segments independently, failing to distinguish between the actual action and similar-looking background noise or "pre-action" movements. To address this, we propose modeling actions as structured processes with distinct phases.
Temporal Structure Mining (TSM)
The core philosophy of TSM is that an action is not a static state but a process with a beginning, middle, and end.
Phase Modeling and Evolution
Instead of a single classifier for an action category, TSM employs multiple "phase filters" ($a_1, a_2, \dots, a_M$) to represent different stages of an activity. To handle untrimmed videos containing multiple action instances, we introduce a background phase ($a_0$) and enforce a circulant phase evolution.
Unlike previous linear models that could only handle a single action per clip, TSM's circulant structure allows the model to cycle from the final action phase back to the background, enabling the detection of multiple instances in a single long video.
Framework Overview and Path Discovery
The TSM framework processes segment features through phase filters to generate a scores table. Since the exact phase of each segment is unknown during training, we treat them as hidden variables.
We use Dynamic Programming to find the "maximal circulant path"—the sequence of phases that maximizes the total score while adhering to the circulant order. To improve training stability and prevent the model from overfitting to noisy segments, we implement a Soft-max Path Discovery variant. This approach considers a weighted ensemble of high-scoring paths rather than just the single best one, providing a richer signal for optimization.
Experimental Results
We evaluated TSM on three major benchmarks: THUMOS14, ActivityNet 1.2, and ActivityNet 1.3. In all experiments, the model was trained using only video-level category labels.
Performance Comparison
TSM significantly outperforms existing state-of-the-art weakly supervised methods. On THUMOS14, TSM achieved a mean Average Precision (mAP) of 24.5% at an Intersection over Union (IoU) threshold of 0.5, a substantial improvement over STPN (16.9%) and Auto-Loc (21.2%). Similar gains were observed on ActivityNet 1.3, where TSM reached an mAP of 19.0% at IoU=0.75.
Key Insights
- Number of Phases: Our analysis showed that using 3 phases provides the best balance between capturing structure and avoiding overfitting. Models using only a single phase performed significantly worse.
- Circulant vs. Linear: The circulant structure was crucial for performance on untrimmed videos, as it naturally handles the transition between multiple action instances and background noise.
Conclusion
Temporal Structure Mining (TSM) demonstrates that even under weak supervision, modeling the internal temporal logic of actions is highly beneficial. By treating actions as multi-phase processes and utilizing a circulant path discovery mechanism, TSM bridges the gap between video-level labels and frame-level localization. Future work will explore adaptive phase counts for different action types and more complex background modeling to further refine detection in cluttered environments.
Summary updated