Towards Understanding Future: Consistency Guided Probabilistic Modeling for Action Anticipation

Zhao Xie, Yadong Shi, Kewei Wu, Yaru Cheng, Dan GuoAAAI 2024 · DOI

Introduction

Predicting human actions before they occur is a fundamental challenge in computer vision, particularly for egocentric (first-person) video applications. While human actions typically follow a logical and continuous flow, existing models often struggle with the inherent uncertainty of the future. A person reaching for a knife might be preparing to cut a vegetable, wash the utensil, or put it away. This paper introduces the Consistency-guided Probabilistic Model (CPM), a framework designed to address "temporal inconsistency" by enforcing a smooth semantic transition between the observed past and the unobserved future.

The Challenge of Temporal Continuity

Current action anticipation models frequently produce "jumpy" or illogical predictions. For instance, a model might observe a user "taking tomatoes" and immediately predict "washing hands," skipping necessary intermediate steps like "cutting tomatoes" simply because "washing hands" is a frequent action in the training data. The authors argue that this occurs because models fail to account for the temporal continuity—the smooth semantic bridge—that connects past observations to future outcomes.

The Consistency-guided Probabilistic Model (CPM)

The CPM framework, built upon a Transformer architecture, utilizes three core modules to ensure that predictions remain contextually grounded and temporally smooth.

Comparison of the proposed CPM against general Transformer-based models, highlighting the focus on temporal continuity and distribution alignment.

1. Future Semantics Estimation

This module generates frame-by-frame predictions of future features. Unlike deterministic models, it employs a "consistency-aware" decoder that incorporates the probabilistic variation of the past to explore a range of likely future trajectories.

2. Global Semantics Estimation

To ensure that the model's understanding of the past aligns with the broader context of the video, the authors introduce a Global Semantics Loss ($L_{sem}$). By using Jaccard vector similarity, the model evaluates the presence of specific action semantics across the entire video segment (past and future), forcing the past features to be consistent with the global context.

3. Global Distribution Estimation

At the heart of the CPM is a probabilistic approach based on Variational Analysis. The model defines two distinct distributions:

  • Past-aware distribution: The predicted future based solely on observed data.
  • Global-aware distribution: The "ideal" ground truth distribution derived from the full video context.
The probabilistic framework of CPM, illustrating how optimizing the Evidence Lower Bound (ELBO) reduces the distance between past-aware and global-aware distributions.

During training, the model minimizes the KL Divergence between these distributions by maximizing the Evidence Lower Bound (ELBO). This process teaches the encoder to produce past-aware predictions that accurately reflect the smooth, logical transitions found in the global reality.

Experimental Results

The CPM was evaluated on three major egocentric video benchmarks, achieving state-of-the-art (SOTA) performance across the board:

  • Epic-Kitchens-100 (EK100): The model achieved a 17.2% Recall@5 using RGB data alone, outperforming established models like DCR-1s and MeMViT. Performance increased to 19.4% when incorporating multi-modal data (Objects and Flow).
  • EGTEA Gaze+: The model reached 50.3% Top-1 Accuracy, marking a significant improvement over previous methods like Abstract Goal (49.8%) and AVT (43.0%).
  • Qualitative Improvements: Visualizations demonstrate that CPM maintains better attention on logical sequences. While baseline models might "hallucinate" premature actions, CPM correctly anticipates the intermediate steps required for a consistent activity flow.

Conclusion

By treating the transition from past to future as a probabilistic distribution rather than a single point, the Consistency-guided Probabilistic Model effectively manages the uncertainty of human behavior. The use of global context as a "teacher" during training allows the model to learn the shape of logical transitions, resulting in more accurate and temporally continuous action anticipation.

Summary updated

BibTeX
@inproceedings{xie2024towards,
  author = {Zhao Xie and Yadong Shi and Kewei Wu and Yaru Cheng and Dan Guo},
  title = {{Towards Understanding Future: Consistency Guided Probabilistic Modeling for Action Anticipation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i6.28442},
}
Figure