Abstract
Localizing actions in video is a core task in computer vision. The weakly supervised temporal localization problem investigates whether this task can be adequately solved with only video-level labels, significantly reducing the amount of expensive and error-prone annotation required. A common approach is to train a frame-level classifier where frames with the highest class probability are selected to make a video-level prediction. However, the absence of frame-level annotations causes the classifier to impart class bias on every frame. To address this, we propose the Action Selection Learning (ASL) approach to capture the general concept of action, a property we refer to as "actionness". Under ASL, the model is trained with a novel class-agnostic task to predict which frames will be selected by the classifier. Empirically, we show that ASL outperforms leading baselines on two popular benchmarks, THUMOS-14 and ActivityNet-1.2, with 10.3% and 5.7% relative improvement, respectively.
Introduction
Temporal Action Localization (TAL) is the task of identifying exactly when an action starts and ends within a video. Performing this "weakly" involves training models using only video-level labels (e.g., "this video contains a cricket shot") without providing specific timestamps. The standard approach utilizes Multiple Instance Learning (MIL), where a classifier selects the "top-k" frames it is most confident about to represent the video.
However, this method frequently encounters two major pitfalls:
- Context Error: The classifier may mistakenly trigger on environmental cues (like a cricket pitch or swimming pool) rather than the action itself.
- Actionness Error: The classifier may ignore difficult or atypical action frames because they do not match the "textbook" visual features of the class.
Methods: Action Selection Learning (ASL)
We propose Action Selection Learning (ASL) to capture "actionness"—the class-agnostic quality of something happening. The ASL framework consists of two main components working in tandem:
- The Base Classifier: A standard network that produces Class Activation Sequences (CAS) to identify what action is occurring.
- The Actionness Model ($G$): A novel, class-agnostic network designed to predict if any action is occurring at all.
The Training Loop
The Actionness Model is trained on a proxy task: predicting which frames the Base Classifier will include in its "top-k" selection. Because context is typically class-specific while the visual signature of action (e.g., movement) is consistent across categories, the Actionness Model learns to ignore static backgrounds and focus on the action.
To mitigate the impact of noisy labels from the classifier during early training, we utilize Generalized Cross Entropy (GCE) loss. This loss function is robust to noise, allowing the model to ignore high-disagreement frames until the classifier becomes more reliable.
Experimental Results
The model was evaluated on two industry-standard benchmarks: THUMOS-14 and ActivityNet-1.2, using I3D features (combining RGB and Flow information).
| Dataset | mAP | Relative Improvement |
|---|---|---|
| THUMOS-14 | 32.2% | 10.3% |
| ActivityNet-1.2 | 25.8% | 5.7% |
Qualitatively, ASL demonstrates superior robustness. In "Cricket Bowling" videos set in atypical indoor environments, ASL successfully identifies the action by relying on learned "actionness" cues rather than expecting a traditional outdoor pitch. This suggests that the model effectively learns to generalize the concept of action beyond specific class contexts.
Conclusion
ASL significantly improves weakly supervised temporal action localization by teaching models to distinguish between the "action" and its "context." By fusing class-specific signals with a universal actionness model, we achieve state-of-the-art performance and more precise localization.
Full code for this work is available at: https://github.com/layer6ai-labs/ASL
Summary updated