Introduction
Action anticipation is a critical task in computer vision, aiming to predict human actions before they occur. While predicting the immediate next action is well-researched, dense action anticipation—forecasting a long sequence of future actions and their specific durations—remains a significant challenge. The primary bottleneck is the requirement for "full supervision," where every frame of a future sequence must be manually labeled with action classes and precise timestamps. This process is labor-intensive and difficult to scale.
To address this, researchers have proposed a weakly-supervised framework that significantly reduces the annotation burden. Instead of labeling the entire future sequence, the model is trained using only the first upcoming action as a "weak label," leaving the subsequent sequence blank.
Methodology: Pseudo-Label and Refine
The proposed framework utilizes a semi-weakly supervised approach centered on a "pseudo-label and refine" strategy. The architecture consists of three primary components:
- Conditional Module: An auxiliary "teacher" model trained on a small subset of fully-labeled data. It generates "hallucinated" pseudo-labels for the unlabeled portions of weakly-labeled videos.
- Primary Module: The main inference model, trained on both the small fully-labeled set and the larger set of pseudo-labeled data.
- Adaptive Refinement Module: To mitigate "confirmation bias"—where the primary model simply mimics the teacher's errors—this module adaptively combines predictions from both modules to generate more accurate training targets.
Furthermore, a Duration Attention mechanism was introduced. Because dense anticipation is highly sensitive to timing, this module computes attention scores between the observed video context and the decoder's hidden states, grounding predicted durations in visual evidence to prevent temporal drift.
Experimental Results
The framework was evaluated on the Breakfast Actions and 50Salads datasets. Experiments utilized only 15-20% of the data as fully labeled, with the remainder treated as weakly labeled.
Key findings include:
- Performance Gains: On the Breakfast dataset (30% observation, 50% prediction), the model achieved a Mean over Classes (MoC) of 8.99, outperforming fully supervised RNN baselines (5.37).
- Annotation Efficiency: Providing "time-stamp" style weak labels was found to be approximately 6x faster for human annotators than marking exact start and end frames for every action.
- Importance of Refinement: The Adaptive Refinement approach outperformed simple linear decay schedules by 4-5% MoC, proving essential for learning from noisy pseudo-labels.
- Feature Impact: Utilizing advanced visual features like I3D significantly boosted performance (21.30 MoC on 50Salads) compared to traditional Fisher vectors (16.24 MoC).
Conclusion
This work demonstrates that dense action anticipation can be effectively performed with significantly reduced human effort. By leveraging a small amount of full supervision to guide the learning of weakly-labeled data through pseudo-labeling and adaptive refinement, the model achieves competitive results. While overall performance in dense anticipation remains an open challenge due to the inherent uncertainty of human activities, this framework provides a scalable path forward for long-term activity forecasting.
Summary updated