Abstract: Training temporal action detection in videos requires large amounts of labeled data, yet such annotation is expensive to collect. Incorporating unlabeled or weakly-labeled data to train action detection model could help reduce annotation cost. In this work, we first introduce the Semi-supervised Action Detection (SSAD) task with a mixture of labeled and unlabeled data and analyze different types of errors in the proposed SSAD baselines which are directly adapted from the semi-supervised classification task. To alleviate the main error of action incompleteness (i.e., missing parts of actions) in SSAD baselines, we further design an unsupervised foreground attention (UFA) module utilizing the "independence" between foreground and background motion. Then we incorporate weakly-labeled data into SSAD and propose Omni-supervised Action Detection (OSAD) with three levels of supervision. An information bottleneck (IB) suppressing the scene information in non-action frames while preserving the action information is designed to help overcome the accompanying action-context confusion problem in OSAD baselines. We extensively benchmark against the baselines for SSAD and OSAD on our created data splits in THUMOS14 and ActivityNet1.2, and demonstrate the effectiveness of the proposed UFA and IB methods. Lastly, the benefit of our full OSAD-IB model under limited annotation budgets is shown by exploring the optimal annotation strategy for labeled, unlabeled and weakly-labeled data.
What Problem Does This Paper Address?
Temporal Action Detection (TAD) is a challenging task in video understanding, requiring models to identify both the class and the precise temporal boundaries (start and end times) of actions. The primary bottleneck for TAD is the high cost of data annotation; labeling precise boundaries across thousands of videos is labor-intensive.
While semi-supervised learning (SSL) has been successful in image classification, directly porting these techniques to video detection reveals two major failure modes:
- Action Incompleteness: Models often fail to capture the full duration of an action, missing the beginning or end.
- Action-Context Confusion: Models often confuse the scene context (e.g., a swimming pool) with the action itself (e.g., swimming), leading to poor localization when the actor is present but not performing the action.
Methods
The authors propose a multi-level supervision framework that leverages a mix of fully labeled, weakly labeled (video-level tags), and unlabeled data to improve performance while reducing costs.
1. Unsupervised Foreground Attention (UFA)
To address action incompleteness, the authors introduce the UFA module. This module exploits the motion independence between the foreground (the actor) and the background (camera motion). By minimizing the conditional mutual information between these two components, the model is forced to focus on the self-contained movement of the actor, helping it identify the full temporal extent of an action without requiring additional boundary labels.
2. Omni-supervised Action Detection (OSAD) & Information Bottleneck (IB)
OSAD incorporates weakly labeled data where only the presence of an action is known, not its timing. To combat action-context confusion, the authors introduce an Information Bottleneck (IB). This module penalizes the model for extracting information from non-action frames, effectively filtering out background scene details that might act as misleading "shortcuts" for the classifier.
Key Results and Insights
The researchers evaluated their methods on the THUMOS14 and ActivityNet1.2 datasets using custom splits to simulate limited annotation budgets.
- Performance Gains: Adding the UFA module to semi-supervised baselines consistently improved mean Average Precision (mAP) on ActivityNet. On THUMOS14, the full OSAD-IB model (using a 10% full, 40% weak, and 50% unlabeled split) achieved a 15.33 [email protected], nearly doubling the 8.08 mAP of a supervised-only model trained on the same 10% of fully labeled data.
- Annotation Efficiency: A key finding is that a "Mixed" annotation strategy is more cost-effective than a "Full-only" strategy. For a fixed budget, combining a small amount of full supervision with a larger amount of weak supervision yielded significantly better results (37.08 vs 34.02 [email protected] on ActivityNet).
- The Role of Context: The study highlights that while context is useful for classification, it can be detrimental to localization. The Information Bottleneck serves as a crucial filter to ensure the model learns the action rather than the environment.
Conclusion
This work demonstrates that multi-level supervision, combined with motion-based attention and information-theoretic bottlenecks, can significantly reduce the dependency on expensive frame-by-frame annotations. By being smarter about how we utilize unlabeled and weakly labeled data, we can build more robust and efficient action detection systems.
Summary updated