Weakly-Supervised Dense Action Anticipation

Haotong Zhang, Fuhai Chen, Angela Yao

Introduction

Action anticipation is a critical task in computer vision, aiming to predict human actions before they occur. While predicting the immediate next action is well-researched, dense action anticipation—forecasting a long sequence of future actions and their specific durations—remains a significant challenge. The primary bottleneck is the requirement for "full supervision," where every frame of a future sequence must be manually labeled with action classes and precise timestamps. This process is labor-intensive and difficult to scale.

To address this, researchers have proposed a weakly-supervised framework that significantly reduces the annotation burden. Instead of labeling the entire future sequence, the model is trained using only the first upcoming action as a "weak label," leaving the subsequent sequence blank.

Comparison between full labels and weak labels for dense action anticipation.

Methodology: Pseudo-Label and Refine

The proposed framework utilizes a semi-weakly supervised approach centered on a "pseudo-label and refine" strategy. The architecture consists of three primary components:

  1. Conditional Module: An auxiliary "teacher" model trained on a small subset of fully-labeled data. It generates "hallucinated" pseudo-labels for the unlabeled portions of weakly-labeled videos.
  2. Primary Module: The main inference model, trained on both the small fully-labeled set and the larger set of pseudo-labeled data.
  3. Adaptive Refinement Module: To mitigate "confirmation bias"—where the primary model simply mimics the teacher's errors—this module adaptively combines predictions from both modules to generate more accurate training targets.
The proposed semi-weakly supervised framework architecture, featuring the Conditional and Primary modules alongside the Sequence Refinement process.

Furthermore, a Duration Attention mechanism was introduced. Because dense anticipation is highly sensitive to timing, this module computes attention scores between the observed video context and the decoder's hidden states, grounding predicted durations in visual evidence to prevent temporal drift.

Experimental Results

The framework was evaluated on the Breakfast Actions and 50Salads datasets. Experiments utilized only 15-20% of the data as fully labeled, with the remainder treated as weakly labeled.

Key findings include:

  • Performance Gains: On the Breakfast dataset (30% observation, 50% prediction), the model achieved a Mean over Classes (MoC) of 8.99, outperforming fully supervised RNN baselines (5.37).
  • Annotation Efficiency: Providing "time-stamp" style weak labels was found to be approximately 6x faster for human annotators than marking exact start and end frames for every action.
  • Importance of Refinement: The Adaptive Refinement approach outperformed simple linear decay schedules by 4-5% MoC, proving essential for learning from noisy pseudo-labels.
  • Feature Impact: Utilizing advanced visual features like I3D significantly boosted performance (21.30 MoC on 50Salads) compared to traditional Fisher vectors (16.24 MoC).

Conclusion

This work demonstrates that dense action anticipation can be effectively performed with significantly reduced human effort. By leveraging a small amount of full supervision to guide the learning of weakly-labeled data through pseudo-labeling and adaptive refinement, the model achieves competitive results. While overall performance in dense anticipation remains an open challenge due to the inherent uncertainty of human activities, this framework provides a scalable path forward for long-term activity forecasting.

Summary updated

Abstract

Dense anticipation aims to forecast future actions and their durations for long horizons. Existing approaches rely on fully-labelled data, i.e. sequences labelled with all future actions and their durations. We present a (semi-) weakly supervised method using only a small number of fully-labelled sequences and predominantly sequences in which only the (one) upcoming action is labelled. To this end, we propose a framework that generates pseudo-labels for future actions and their durations and adaptively refines them through a refinement module. Given only the upcoming action label as input, these pseudo-labels guide action/duration prediction for the future. We further design an attention mechanism to predict context-aware durations. Experiments on the Breakfast and 50Salads benchmarks verify our method's effectiveness; we are competitive even when compared to fully supervised state-of-the-art models. We will make our code available at: https://github.com/zhanghaotong1/WSLVideoDenseAnticipation.
BibTeX
@inproceedings{zhang2021weakly,
  author = {Haotong Zhang and Fuhai Chen and Angela Yao},
  title = {{Weakly-Supervised Dense Action Anticipation}},
  booktitle = {BMVC},
  year = {2021},
  url = {https://openreview.net/forum?id=lQNaEcnkUu},
}
Figure