Enhancing Next Active Object-Based Egocentric Action Anticipation with Guided Attention

Sanket Kumar Thakur, Cigdem Beyan, Pietro Morerio, Vittorio Murino, Alessio Del Bue

Introduction

Short-term action anticipation (STA) is a fundamental challenge in egocentric (first-person) vision, with significant implications for the development of Augmented Reality (AR) systems and human-robot interaction. Unlike third-person video analysis, egocentric vision must contend with constant camera motion, frequent object occlusions, and the dynamic nature of the wearer's environment. Traditional approaches to STA often attempt to predict future actions by analyzing video clips holistically. However, this paper argues that such methods overlook a critical contextual anchor: the Next Active Object (NAO).

Identifying the specific object a user is about to interact with—such as a mug before it is grasped or a handle before it is turned—drastically constrains the search space for potential actions. By focusing on the NAO, models can more accurately predict not just the action (verb), but also the spatial coordinates (bounding box) and the precise timing (Time to Contact) of the interaction.

The GANO Framework

The authors propose GANO (Guided Attention for Next active Objects), a transformer-based framework designed to integrate motion and object-centric data through a novel attention mechanism. The architecture operates in three distinct stages:

1. Feature Extraction

The model processes video input through two parallel streams:

  • Spatiotemporal Patches: A 3D Convolutional (Conv3D) layer extracts motion-based features from the video sequence.
  • Object Embeddings: A pre-trained object detector identifies candidate objects within the frames. These candidates are processed via a Multi-Layer Perceptron (MLP) to create high-dimensional object embeddings.

2. Guided Attention Mechanism

The core innovation of GANO is the Guided Attention module. Rather than simply concatenating video and object features—a method the authors found to be suboptimal—GANO utilizes a modified Multi-Head Attention (MHA) mechanism. In this setup, the spatiotemporal video patches serve as the Query (Q), while the object embeddings serve as the Key (K) and Value (V). This configuration "guides" the model to prioritize motion features that are spatially and semantically relevant to the objects in the scene, effectively learning the relationship between the actor's intent and the physical environment.

3. Transformer Encoder-Decoder

The fused features are processed by an encoder utilizing Multi-Scale Vision Transformer (MViT) blocks to capture long-term temporal dependencies. The decoder then uses Regions of Interest (ROIs) from the final observed frame as queries to generate four specific outputs: the object's noun label, its predicted bounding box, the future verb, and the Time to Contact (TTC).

Experimental Results

GANO was evaluated on the Ego4D dataset, the largest existing repository of egocentric video, specifically utilizing the forecasting split which contains over 960 hours of annotated interactions.

Performance Metrics

GANO achieved state-of-the-art (SOTA) results across multiple benchmarks, outperforming established models such as SlowFast, AVT, and MeMViT. On the primary metric, $AP_{b+\hat{v}+\delta}$—which requires the simultaneous correct prediction of the bounding box, verb, and TTC—GANO reached a score of 2.77, a significant improvement over MeMViT’s 2.11.

Key Insights

  • Ablation Studies: Removing the guided attention mechanism resulted in a drastic performance drop, confirming that the interaction between motion and object features is the primary driver of the model's success.
  • Temporal Precision: GANO showed marked improvements in predicting Time to Contact (TTC), suggesting that the guided attention mechanism allows the model to better perceive the spatial velocity and distance of the wearer's hands relative to target objects.
  • Discovery via Learnable Tokens: By appending learnable object tokens to the ROI embeddings in the decoder, the model demonstrated an improved ability to identify the next active object even when the initial object detector failed to register it.

Conclusion and Future Work

GANO demonstrates that centering action anticipation on the "Next Active Object" provides a more robust framework for understanding human intent in egocentric video. While the model sets a new benchmark on Ego4D, the authors noted a slight trade-off: optimizing heavily for object location and noun accuracy can occasionally lead to a minor decrease in verb prediction accuracy.

Future research will focus on balancing these multi-task losses and exploring the application of GANO in real-time human-robot collaboration and automated video summarization for long-form egocentric recordings.

Summary updated

BibTeX
@inproceedings{thakur2023enhancing,
  author = {Sanket Kumar Thakur and Cigdem Beyan and Pietro Morerio and Vittorio Murino and Alessio Del Bue},
  title = {{Enhancing Next Active Object-Based Egocentric Action Anticipation with Guided Attention}},
  booktitle = {ICIP},
  year = {2023},
  doi = {10.1109/icip49359.2023.10222445},
}
Figure