Learning to Anticipate Egocentric Actions by Imagination

Yu Wu, Linchao Zhu, Xiaohan Wang, Yi Yang, Fei Wu

TL;DR

This paper introduces ImagineRNN, a framework that improves egocentric action anticipation by "imagining" the visual features of the unobserved temporal gap between the present and the future. By combining contrastive learning with residual feature prediction, the model effectively bridges the gap between past observations and future actions, setting new state-of-the-art results on major benchmarks like EPIC-KITCHENS.

What Problem Does This Paper Address?

In egocentric (first-person) video analysis, action anticipation is the task of predicting what a person will do seconds before they actually do it (e.g., predicting "wash pan" while the person is still walking toward the sink).

The core challenge is the temporal gap. Most existing models summarize past observations and jump straight to a future prediction. However, the frames that occur during the anticipation window—which are unobserved at the time of prediction—contain the most critical cues for the upcoming action. The authors argue that instead of ignoring this gap, models should attempt to "fill it in" by forecasting how visual features will evolve.

Methods

The authors propose a "Learning by Imagination" approach that decomposes long-term anticipation into a series of short-term feature forecasts.

  1. ImagineRNN: This is the heart of the system. It is a recurrent neural network (LSTM) designed to predict the next visual representation based on past states. Instead of predicting raw pixels (which is computationally expensive and noisy), it predicts high-level features.
  2. Contrastive Learning (NCE): Standard feature regression (like L2 loss) often results in "blurry" or averaged features because the future is inherently uncertain. To solve this, the authors use Noise Contrastive Estimation (NCE). The model is trained to pick the "correct" future feature out of a lineup of "distractors" (features from different time steps or different videos). This forces the model to learn more discriminative and meaningful transitions.
  3. Residual Anticipation: To prevent the model from wasting capacity on static backgrounds, the authors task ImagineRNN with predicting the feature difference (the residual) between adjacent frames rather than the entire frame content. This acts as a shortcut connection, focusing the model's "imagination" on the dynamic changes relevant to the action.
  4. Encoder-Decoder Architecture: An encoder processes the observed video, ImagineRNN generates the intermediate "imagined" features, and a decoder uses both to produce the final action label (Verb + Noun).

Datasets

The model was evaluated on two large-scale egocentric benchmarks:

  • EPIC-KITCHENS: The largest dataset in the field, featuring 55 hours of non-scripted kitchen activities. It includes a "Seen" test set (S1) and an "Unseen" test set (S2) featuring entirely different kitchens.
  • EGTEA Gaze+: A smaller but highly controlled dataset featuring 106 unique actions with gaze tracking information.

Key Insights and Ideas

  • Bridging the Gap: Decomposing a 1-second jump into four 0.25-second "imagination" steps makes the learning process more stable and accurate.
  • The Power of Contrast: Using a proxy task (picking the right future state from distractors) is significantly more effective for generalizability than simple feature regression. This is particularly evident on the "Unseen" test sets, where the model must adapt to new environments.
  • Focusing on Dynamics: By predicting residuals (changes) rather than full features, the model avoids noise accumulation and focuses on the "core" of the action—the movement of hands and objects.

Key Results

The proposed method outperformed previous state-of-the-art models (like RULSTM) across all metrics:

  • EPIC-KITCHENS (Seen): Achieved 34.98% Top-5 Action Accuracy, a notable improvement over the previous best of 33.73%.
  • EPIC-KITCHENS (Unseen): The gap was even wider here, with the contrastive "imagination" approach reaching 22.19% Top-5 Action Accuracy compared to 21.10% for RULSTM, proving the method's robustness to new environments.
  • Ablation Studies: Removing the "imagination" component (ImagineRNN) caused a significant drop in performance, confirming that filling the temporal gap is essential. Using contrastive learning instead of L2 loss provided a ~1.4% boost on unseen data.

Limitations and Future Work

  • Future Uncertainty: The current model predicts a single deterministic path for the future. However, human actions are often multi-modal (a person reaching for a counter might grab a knife or a spoon). The authors suggest that explicitly modeling this uncertainty is a key next step.
  • Fine-grained Cues: While the model uses global features, the authors note that incorporating specific hand-tracking or object-interaction cues could further refine the "imagined" intermediate states.
  • Computational Load: While more efficient than pixel-level generation, the auto-regressive nature of ImagineRNN adds some computational overhead during inference.

Summary updated

Abstract

Anticipating actions before they are executed is crucial for a wide range of practical applications, including autonomous driving and robotics. In this paper, we study the egocentric action anticipation task, which predicts future action seconds before it is performed for egocentric videos. Previous approaches focus on summarizing the observed content and directly predicting future action based on past observations. We believe it would benefit the action anticipation if we could mine some cues to compensate for the missing information of the unobserved frames. We then propose to decompose the action anticipation into a series of future feature predictions. We imagine how the visual feature changes in the near future and then predicts future action labels based on these imagined representations. Differently, our ImagineRNN is optimized in a contrastive learning way instead of feature regression. We utilize a proxy task to train the ImagineRNN, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , selecting the correct future states from distractors. We further improve ImagineRNN by residual anticipation, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , changing its target to predicting the feature difference of adjacent frames instead of the frame content. This promotes the network to focus on our target, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , the future action, as the difference between adjacent frame features is more important for forecasting the future. Extensive experiments on two large-scale egocentric action datasets validate the effectiveness of our method. Our method significantly outperforms previous methods on both the seen test set and the unseen test set of the EPIC Kitchens Action Anticipation Challenge.
BibTeX
@article{wu2021learning,
  author = {Yu Wu and Linchao Zhu and Xiaohan Wang and Yi Yang and Fei Wu},
  title = {{Learning to Anticipate Egocentric Actions by Imagination}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=fp3qDmEtZt2},
}

Loading related content...

Figure