Abstract
Objects are crucial for understanding human-object interactions. By identifying the relevant objects, one can also predict potential future interactions or actions that may occur with these objects. In this paper, we study the problem of Short-Term Object interaction anticipation (STA) and propose NAOGAT (Next-Active-Object Guided Anticipation Transformer), a multi-modal end-to-end transformer network, that attends to objects in observed frames in order to anticipate the next-active-object (NAO) and, eventually, to guide the model to predict context-aware future actions. The task is challenging since it requires anticipating future action along with the object with which the action occurs and the time after which the interaction will begin, a.k.a. the time to contact (TTC). Compared to existing video modeling architectures for action anticipation, NAOGAT captures the relationship between objects and the global scene context in order to predict detections for the next active object and anticipate relevant future actions given these detections, leveraging the objects' dynamics to improve accuracy. One of the key strengths of our approach is its ability to exploit the motion dynamics of objects within a given clip, which is often ignored by other models, and separately decoding the object-centric and motion-centric information. Through our experiments, we show that our model outperforms existing methods on two separate datasets, Ego4D and EpicKitchens-100 ("Unseen Set"), as measured by several additional metrics, such as time to contact, and next-active-object localization.
Introduction
In the rapidly evolving fields of Augmented Reality (AR) and robotics, systems must move beyond simple recognition to proactive anticipation. Short-Term Anticipation (STA) is the ability of a model to predict what interaction will occur next, which object will be involved, and exactly when that interaction will begin. This is a complex task in egocentric (first-person) video, where human actions are inextricably linked to the surrounding environment.
Most current models rely on holistic scene features or hand-tracking to predict future actions. However, these approaches often overlook the specific dynamics of the objects themselves and the "Time to Contact" (TTC)—the precise temporal gap before an interaction starts. The authors of this paper identify a critical gap: existing architectures do not effectively use objects as "contextual anchors" to guide their predictions of future verbs and timing. To address this, they introduce NAOGAT, which explicitly identifies the "Next-Active Object" (NAO) to refine its understanding of the scene.
Methodology: The NAOGAT Architecture
The proposed NAOGAT (Next-Active-Object Guided Anticipation Transformer) is a multi-modal framework designed to separate and then integrate object-centric and motion-centric information. The architecture is composed of four primary blocks:
- Multimodal Encoding: The system processes a sequence of video frames using a Video Swin Transformer backbone to extract rich visual features. Concurrently, a Faster R-CNN detector identifies objects within the scene. These RGB features and object detections are fused to create a comprehensive scene representation.
- The NAO Block: This specialized transformer decoder uses "object queries" derived from the initial detections. Its goal is to pinpoint which object in the last observed frame is the most likely candidate for the next interaction, predicting its noun and bounding box.
- The Motion Block: This block handles the temporal dynamics of the scene. It distinguishes between Foreground Motion (e.g., the user's hand movements) and Background/Object Dynamics (the trajectories of objects across the clip). This separation allows the model to treat object trajectories as a proxy for the physics of the scene.
- Joint Prediction: By synthesizing the identity of the NAO with these dual motion cues, the model outputs the future verb and the Time to Contact (TTC).
Datasets and Evaluation
NAOGAT was evaluated on two benchmark egocentric datasets:
- Ego4D: The largest first-person video dataset, specifically the "forecasting split" for STA tasks.
- EpicKitchens-100 (EK-100): A standard for action recognition. Since EK-100 lacked "Next-Active Object" annotations, the authors curated a new set of labels using a hand-object detector pipeline to identify objects just before contact, providing a new resource for the community.
Key Results and Insights
The experimental results demonstrate that NAOGAT establishes a new state-of-the-art for short-term anticipation. On the Ego4D dataset, the model achieved a 2.16% gain in Average Precision (AP) for the "Verb + NAO" task. Most significantly, it showed a 7.33% improvement in estimating the "TTC + NAO" combination compared to previous leading models like MeMViT and AVT.
Beyond raw metrics, qualitative analysis reveals that the NAO decoder often improves upon the initial object detector. By leveraging temporal context, it can localize objects that a static detector might miss. Furthermore, the model successfully ignores passive background objects to focus on the specific tools or items the user is about to engage with.
Conclusion and Future Work
NAOGAT demonstrates that identifying the Next-Active Object is not merely a sub-task but a prerequisite for accurate action anticipation. By treating objects as contextual anchors and modeling their motion dynamics, the system achieves superior precision in both action identity and timing.
A primary limitation remains the "Visibility Assumption"—the model currently struggles when the NAO is not visible in the final observed frame (e.g., an object hidden in a drawer). Future research will focus on integrating object trackers to handle temporary occlusions and incorporating human-centric cues like eye-gaze data to further refine the understanding of user intent.
Summary updated