Introduction
Predicting human actions before they occur is a fundamental challenge in computer vision, particularly for egocentric (first-person) video applications. While human actions typically follow a logical and continuous flow, existing models often struggle with the inherent uncertainty of the future. A person reaching for a knife might be preparing to cut a vegetable, wash the utensil, or put it away. This paper introduces the Consistency-guided Probabilistic Model (CPM), a framework designed to address "temporal inconsistency" by enforcing a smooth semantic transition between the observed past and the unobserved future.
The Challenge of Temporal Continuity
Current action anticipation models frequently produce "jumpy" or illogical predictions. For instance, a model might observe a user "taking tomatoes" and immediately predict "washing hands," skipping necessary intermediate steps like "cutting tomatoes" simply because "washing hands" is a frequent action in the training data. The authors argue that this occurs because models fail to account for the temporal continuity—the smooth semantic bridge—that connects past observations to future outcomes.
The Consistency-guided Probabilistic Model (CPM)
The CPM framework, built upon a Transformer architecture, utilizes three core modules to ensure that predictions remain contextually grounded and temporally smooth.
1. Future Semantics Estimation
This module generates frame-by-frame predictions of future features. Unlike deterministic models, it employs a "consistency-aware" decoder that incorporates the probabilistic variation of the past to explore a range of likely future trajectories.
2. Global Semantics Estimation
To ensure that the model's understanding of the past aligns with the broader context of the video, the authors introduce a Global Semantics Loss ($L_{sem}$). By using Jaccard vector similarity, the model evaluates the presence of specific action semantics across the entire video segment (past and future), forcing the past features to be consistent with the global context.
3. Global Distribution Estimation
At the heart of the CPM is a probabilistic approach based on Variational Analysis. The model defines two distinct distributions:
- Past-aware distribution: The predicted future based solely on observed data.
- Global-aware distribution: The "ideal" ground truth distribution derived from the full video context.
During training, the model minimizes the KL Divergence between these distributions by maximizing the Evidence Lower Bound (ELBO). This process teaches the encoder to produce past-aware predictions that accurately reflect the smooth, logical transitions found in the global reality.
Experimental Results
The CPM was evaluated on three major egocentric video benchmarks, achieving state-of-the-art (SOTA) performance across the board:
- Epic-Kitchens-100 (EK100): The model achieved a 17.2% Recall@5 using RGB data alone, outperforming established models like DCR-1s and MeMViT. Performance increased to 19.4% when incorporating multi-modal data (Objects and Flow).
- EGTEA Gaze+: The model reached 50.3% Top-1 Accuracy, marking a significant improvement over previous methods like Abstract Goal (49.8%) and AVT (43.0%).
- Qualitative Improvements: Visualizations demonstrate that CPM maintains better attention on logical sequences. While baseline models might "hallucinate" premature actions, CPM correctly anticipates the intermediate steps required for a consistent activity flow.
Conclusion
By treating the transition from past to future as a probabilistic distribution rather than a single point, the Consistency-guided Probabilistic Model effectively manages the uncertainty of human behavior. The use of global context as a "teacher" during training allows the model to learn the shape of logical transitions, resulting in more accurate and temporally continuous action anticipation.
Summary updated