Abstract
Anticipating actions before they are executed is crucial for a wide range of practical applications, including autonomous driving and robotics. In this paper, we study the egocentric action anticipation task, which predicts future action seconds before it is performed for egocentric videos. Previous approaches focus on summarizing the observed content and directly predicting future action based on past observations. We believe it would benefit the action anticipation if we could mine some cues to compensate for the missing information of the unobserved frames. We then propose to decompose the action anticipation into a series of future feature predictions. We imagine how the visual feature changes in the near future and then predicts future action labels based on these imagined representations. Differently, our ImagineRNN is optimized in a contrastive learning way instead of feature regression. We utilize a proxy task to train the ImagineRNN, i.e., selecting the correct future states from distractors. We further improve ImagineRNN by residual anticipation, i.e., changing its target to predicting the feature difference of adjacent frames instead of the frame content. This promotes the network to focus on our target, i.e., the future action, as the difference between adjacent frame features is more important for forecasting the future. Extensive experiments on two large-scale egocentric action datasets validate the effectiveness of our method. Our method significantly outperforms previous methods on both the seen test set and the unseen test set of the EPIC Kitchens Action Anticipation Challenge.
Introduction
In egocentric (first-person) video analysis, action anticipation is the task of predicting what a person will do seconds before they actually do it—for example, predicting "wash pan" while the person is still walking toward the sink. The core challenge is the temporal gap. Most existing models summarize past observations and jump straight to a future prediction. However, the frames that occur during the anticipation window—which are unobserved at the time of prediction—contain the most critical cues for the upcoming action.
Instead of ignoring this gap, this paper proposes a "Learning by Imagination" approach that decomposes long-term anticipation into a series of short-term feature forecasts, effectively bridging the gap between past observations and future actions.
Methodology
The proposed framework utilizes a specialized recurrent neural network to fill the unobserved temporal gap with predicted visual features.
ImagineRNN and Residual Anticipation
ImagineRNN is the heart of the system, designed to predict the next visual representation based on past states. Instead of predicting raw pixels, which is computationally expensive and noisy, it predicts high-level features. To prevent the model from wasting capacity on static backgrounds, ImagineRNN is tasked with predicting the feature difference (the residual) between adjacent frames rather than the entire frame content. This acts as a shortcut connection, focusing the model's "imagination" on the dynamic changes relevant to the action.
Contrastive Learning (NCE)
Standard feature regression (like L2 loss) often results in "blurry" or averaged features because the future is inherently uncertain. To solve this, the authors use Noise Contrastive Estimation (NCE). The model is trained to pick the "correct" future feature out of a lineup of "distractors" (features from different time steps or different videos). This forces the model to learn more discriminative and meaningful transitions, which is particularly effective for generalizability on unseen environments.
Experiments and Results
The model was evaluated on two large-scale egocentric benchmarks: EPIC-KITCHENS (55 hours of non-scripted kitchen activities) and EGTEA Gaze+.
Key Performance Metrics
The proposed method outperformed previous state-of-the-art models (like RULSTM) across all metrics:
- EPIC-KITCHENS (Seen): Achieved 34.98% Top-5 Action Accuracy, an improvement over the previous best of 33.73%.
- EPIC-KITCHENS (Unseen): The gap was even wider, reaching 22.19% Top-5 Action Accuracy compared to 21.10% for RULSTM, proving the method's robustness to new environments.
Ablation Insights
Ablation studies confirmed that removing the "imagination" component (ImagineRNN) caused a significant drop in performance. Furthermore, using contrastive learning instead of L2 loss provided a ~1.4% boost on unseen data, validating the proxy task's effectiveness.
Conclusion
By decomposing action anticipation into a series of imagined intermediate states, ImagineRNN effectively addresses the temporal gap inherent in egocentric video. While the current model predicts a single deterministic path, future work may involve explicitly modeling the multi-modal uncertainty of human actions and incorporating fine-grained cues like hand-tracking to further refine the imagined states.
Summary updated