Leveraging Next-Active Objects for Context-Aware Anticipation in Egocentric Videos

Sanket Kumar Thakur, Cigdem Beyan, Pietro Morerio, Vittorio Murino, Alessio Del Bue

Introduction

Predicting human-object interactions in first-person (egocentric) videos is a cornerstone for developing intelligent AR assistants and autonomous robots. Unlike standard action recognition, Short-Term Object interaction anticipation (STA) requires a system to predict not just the action ("what"), but also the specific target object ("where") and the precise time to contact ("when"). This task is inherently difficult due to the shaky, cluttered nature of egocentric footage and the dynamic movement of both the camera and the objects.

The NAOGAT Architecture

To address these challenges, the researchers developed NAOGAT (Next-Active-Object Guided Anticipation Transformer). This multi-modal architecture treats anticipation as a structured prediction problem, moving beyond holistic scene features to focus on the specific dynamics of objects about to be engaged.

The NAOGAT architecture consists of a feature extractor, a transformer encoder, and specialized decoders for object identification (NAO Block) and motion prediction (Motion Block).

The pipeline is divided into four primary components:

  1. Feature Extractor: Employs a Video Swin Transformer (Swin-T) for visual features and a Faster R-CNN for initial object detection.
  2. Transformer Encoder: Fuses visual features with object embeddings (location and confidence scores) to build a comprehensive "video memory."
  3. NAO Block (Object Decoder): Inspired by the Detection Transformer (DETR), this block uses object queries to identify the Next-Active-Object (NAO), refining its bounding box and class label based on the global context.
  4. Motion Block (Motion Decoder): This component extracts "Object Dynamics"—the trajectories of objects across frames—to model background motion. By combining this with foreground motion from the encoder, the model predicts the future Verb and the Time to Contact (TTC).

Experimental Results

The model was rigorously evaluated on two major benchmarks: Ego4D, the largest first-person dataset, and EpicKitchens-100 (EK-100). For the latter, the authors contributed new NAO location annotations to the research community to facilitate future benchmarking.

Quantitative comparison on the Ego4D dataset. NAOGAT (Ours) demonstrates significant improvements in Average Precision (AP) across multiple interaction metrics compared to existing state-of-the-art models.

NAOGAT achieved state-of-the-art results, notably outperforming previous methods on the Ego4D forecasting split. Key performance highlights include:

  • A 2.16% gain in Average Precision (AP) for combined Verb + NAO prediction ($AP_{\hat{b}+\hat{n}+\hat{v}}$).
  • A substantial 7.33% improvement in estimating the joint TTC + NAO metric ($AP_{\hat{b}+\delta}$).
  • Superior generalization on the EK-100 "Unseen Set," indicating robustness across different environments.

Furthermore, NAOGAT is highly efficient. With only 23.5 million parameters, it is significantly more lightweight than competing models like AVT, which utilizes 393 million parameters, making it more suitable for real-time applications.

Conclusion and Future Work

By anchoring action prediction in the dynamics of the Next-Active-Object, NAOGAT provides a more context-aware approach to egocentric anticipation. While the model currently assumes the target object is visible in the final observed frame, future iterations could incorporate object tracking or gaze cues to anticipate interactions with objects that are currently occluded or out of view.

Summary updated

Abstract

Objects are crucial for understanding human-object interactions. By identifying the relevant objects, one can also predict potential future interactions or actions that may occur with these objects. In this paper, we study the problem of Short-Term Object interaction anticipation (STA) and propose NAOGAT (Next-Active-Object Guided Anticipation Transformer), a multi-modal end-to-end transformer network, that attends to objects in observed frames in order to anticipate the next-active-object (NAO) and, eventually, to guide the model to predict context-aware future actions. The task is challenging since it requires anticipating future action along with the object with which the action occurs and the time after which the interaction will begin, a.k.a. the time to contact (TTC). Compared to existing video modeling architectures for action anticipation, NAOGAT captures the relationship between objects and the global scene context in order to predict detections for the next active object and anticipate relevant future actions given these detections, leveraging the objects’ dynamics to improve accuracy. One of the key strengths of our approach, in fact, is its ability to exploit the motion dynamics of objects within a given clip , which is often ignored by other models, and separately decoding the object-centric and motion-centric information. Through our experiments, we show that our model outperforms existing methods on two separate datasets, Ego4D and EpicKitchens-100 ("Unseen Set"), as measured by several additional metrics, such as time to contact, and next-active-object localization. The code can be found on project page : sanketsans.github.io/wacv24
BibTeX
@inproceedings{thakur2024leveraging,
  author = {Sanket Kumar Thakur and Cigdem Beyan and Pietro Morerio and Vittorio Murino and Alessio Del Bue},
  title = {{Leveraging Next-Active Objects for Context-Aware Anticipation in Egocentric Videos}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00846},
}
Figure