Introduction
Predicting human-object interactions in first-person (egocentric) videos is a cornerstone for developing intelligent AR assistants and autonomous robots. Unlike standard action recognition, Short-Term Object interaction anticipation (STA) requires a system to predict not just the action ("what"), but also the specific target object ("where") and the precise time to contact ("when"). This task is inherently difficult due to the shaky, cluttered nature of egocentric footage and the dynamic movement of both the camera and the objects.
The NAOGAT Architecture
To address these challenges, the researchers developed NAOGAT (Next-Active-Object Guided Anticipation Transformer). This multi-modal architecture treats anticipation as a structured prediction problem, moving beyond holistic scene features to focus on the specific dynamics of objects about to be engaged.
The pipeline is divided into four primary components:
- Feature Extractor: Employs a Video Swin Transformer (Swin-T) for visual features and a Faster R-CNN for initial object detection.
- Transformer Encoder: Fuses visual features with object embeddings (location and confidence scores) to build a comprehensive "video memory."
- NAO Block (Object Decoder): Inspired by the Detection Transformer (DETR), this block uses object queries to identify the Next-Active-Object (NAO), refining its bounding box and class label based on the global context.
- Motion Block (Motion Decoder): This component extracts "Object Dynamics"—the trajectories of objects across frames—to model background motion. By combining this with foreground motion from the encoder, the model predicts the future Verb and the Time to Contact (TTC).
Experimental Results
The model was rigorously evaluated on two major benchmarks: Ego4D, the largest first-person dataset, and EpicKitchens-100 (EK-100). For the latter, the authors contributed new NAO location annotations to the research community to facilitate future benchmarking.
NAOGAT achieved state-of-the-art results, notably outperforming previous methods on the Ego4D forecasting split. Key performance highlights include:
- A 2.16% gain in Average Precision (AP) for combined Verb + NAO prediction ($AP_{\hat{b}+\hat{n}+\hat{v}}$).
- A substantial 7.33% improvement in estimating the joint TTC + NAO metric ($AP_{\hat{b}+\delta}$).
- Superior generalization on the EK-100 "Unseen Set," indicating robustness across different environments.
Furthermore, NAOGAT is highly efficient. With only 23.5 million parameters, it is significantly more lightweight than competing models like AVT, which utilizes 393 million parameters, making it more suitable for real-time applications.
Conclusion and Future Work
By anchoring action prediction in the dynamics of the Next-Active-Object, NAOGAT provides a more context-aware approach to egocentric anticipation. While the model currently assumes the target object is visible in the final observed frame, future iterations could incorporate object tracking or gaze cues to anticipate interactions with objects that are currently occluded or out of view.
Summary updated