TL;DR
This paper introduces ImagineRNN, a framework that improves egocentric action anticipation by "imagining" the visual features of the unobserved temporal gap between the present and the future. By combining contrastive learning with residual feature prediction, the model effectively bridges the gap between past observations and future actions, setting new state-of-the-art results on major benchmarks like EPIC-KITCHENS.
What Problem Does This Paper Address?
In egocentric (first-person) video analysis, action anticipation is the task of predicting what a person will do seconds before they actually do it (e.g., predicting "wash pan" while the person is still walking toward the sink).
The core challenge is the temporal gap. Most existing models summarize past observations and jump straight to a future prediction. However, the frames that occur during the anticipation window—which are unobserved at the time of prediction—contain the most critical cues for the upcoming action. The authors argue that instead of ignoring this gap, models should attempt to "fill it in" by forecasting how visual features will evolve.
Methods
The authors propose a "Learning by Imagination" approach that decomposes long-term anticipation into a series of short-term feature forecasts.
- ImagineRNN: This is the heart of the system. It is a recurrent neural network (LSTM) designed to predict the next visual representation based on past states. Instead of predicting raw pixels (which is computationally expensive and noisy), it predicts high-level features.
- Contrastive Learning (NCE): Standard feature regression (like L2 loss) often results in "blurry" or averaged features because the future is inherently uncertain. To solve this, the authors use Noise Contrastive Estimation (NCE). The model is trained to pick the "correct" future feature out of a lineup of "distractors" (features from different time steps or different videos). This forces the model to learn more discriminative and meaningful transitions.
- Residual Anticipation: To prevent the model from wasting capacity on static backgrounds, the authors task ImagineRNN with predicting the feature difference (the residual) between adjacent frames rather than the entire frame content. This acts as a shortcut connection, focusing the model's "imagination" on the dynamic changes relevant to the action.
- Encoder-Decoder Architecture: An encoder processes the observed video, ImagineRNN generates the intermediate "imagined" features, and a decoder uses both to produce the final action label (Verb + Noun).
Datasets
The model was evaluated on two large-scale egocentric benchmarks:
- EPIC-KITCHENS: The largest dataset in the field, featuring 55 hours of non-scripted kitchen activities. It includes a "Seen" test set (S1) and an "Unseen" test set (S2) featuring entirely different kitchens.
- EGTEA Gaze+: A smaller but highly controlled dataset featuring 106 unique actions with gaze tracking information.
Key Insights and Ideas
- Bridging the Gap: Decomposing a 1-second jump into four 0.25-second "imagination" steps makes the learning process more stable and accurate.
- The Power of Contrast: Using a proxy task (picking the right future state from distractors) is significantly more effective for generalizability than simple feature regression. This is particularly evident on the "Unseen" test sets, where the model must adapt to new environments.
- Focusing on Dynamics: By predicting residuals (changes) rather than full features, the model avoids noise accumulation and focuses on the "core" of the action—the movement of hands and objects.
Key Results
The proposed method outperformed previous state-of-the-art models (like RULSTM) across all metrics:
- EPIC-KITCHENS (Seen): Achieved 34.98% Top-5 Action Accuracy, a notable improvement over the previous best of 33.73%.
- EPIC-KITCHENS (Unseen): The gap was even wider here, with the contrastive "imagination" approach reaching 22.19% Top-5 Action Accuracy compared to 21.10% for RULSTM, proving the method's robustness to new environments.
- Ablation Studies: Removing the "imagination" component (ImagineRNN) caused a significant drop in performance, confirming that filling the temporal gap is essential. Using contrastive learning instead of L2 loss provided a ~1.4% boost on unseen data.
Limitations and Future Work
- Future Uncertainty: The current model predicts a single deterministic path for the future. However, human actions are often multi-modal (a person reaching for a counter might grab a knife or a spoon). The authors suggest that explicitly modeling this uncertainty is a key next step.
- Fine-grained Cues: While the model uses global features, the authors note that incorporating specific hand-tracking or object-interaction cues could further refine the "imagined" intermediate states.
- Computational Load: While more efficient than pixel-level generation, the auto-regressive nature of ImagineRNN adds some computational overhead during inference.
Summary updated