Introduction
Egocentric Action Anticipation involves predicting a user's future actions from a first-person perspective—for instance, anticipating that a user currently "stirring chicken" will likely "cut bell peppers" next. This task is inherently challenging due to the temporal gap between observation and action, as well as the high uncertainty of future events. Traditional models often rely solely on visual cues and standard Recurrent Neural Networks (RNNs), leading to a performance plateau. The VS-TransGRU framework addresses these limitations by integrating high-level semantic context and a hybrid architecture to model long-term dependencies effectively.
Proposed Framework: VS-TransGRU
The VS-TransGRU (Visual-Semantic Transformer-GRU) framework introduces three primary innovations to overcome the limitations of visual-only models.
1. Visual-Semantic Fusion
The model augments raw visual features with semantic information. Two variants are proposed:
- Ground Truth Semantic (GTS): Utilizes actual text labels processed via a pre-trained language model (Phrase-Bert) during training to provide a perfect semantic anchor.
- Estimated Semantic (ES): A practical version that learns to estimate semantic features from visual observations using a weighted sum of category embeddings.
2. Hybrid Transformer-GRU Architecture
The architecture leverages a Transformer Encoder for global sequential modeling of the observation period, capturing complex dependencies across time. This is followed by a GRU Decoder, which iteratively handles the anticipation phase, allowing for flexible predictions across various future time horizons (e.g., 0.25s to 2.0s).
3. Learnable Fusion Module
A learnable weighted sum is employed to combine visual and semantic data, allowing the model to dynamically adjust its reliance on each modality for optimal performance.
Evaluation and Results
The framework was evaluated on the EPIC-Kitchens and EGTEA Gaze+ datasets, which are large-scale benchmarks for unscripted real-world activities.
Key results on the EPIC-Kitchens dataset include:
- VS-TransGRU:GTS achieved a 40.06% Top-5 Action Accuracy at a 1-second anticipation time, representing a massive 10.26% improvement over the previous best-performing method (SRL).
- VS-TransGRU:ES (the practical version) reached 32.36% accuracy, outperforming the state-of-the-art by 1.02%.
- The model demonstrated robust and consistent performance improvements across all tested anticipation intervals, from 0.25s to 2.0s.
Conclusion and Future Work
VS-TransGRU demonstrates that bridging the gap between raw pixels and semantic concepts significantly enhances action anticipation. While the model sets new benchmarks, challenges remain in handling ambiguous contexts and camera occlusions where target objects are not yet visible. Future research may explore incorporating additional modalities like audio or more sophisticated human-object interaction modeling to further refine predictive accuracy in complex environments.
Summary updated