Abstract
Temporal Interaction Localization (TIL) in egocentric videos is critical for applications in robotics and AR/VR, yet remains challenging due to the need for fine-grained precision and 3D spatial awareness. We present EgoLoc, a zero-shot framework that localizes hand-object contact and separation moments. EgoLoc leverages 3D hand-motion dynamics for self-adaptive sampling and a closed-loop reasoning system powered by Vision-Language Models (VLMs). By incorporating physical priors and a feedback mechanism that utilizes negative in-context learning, EgoLoc achieves high-precision localization without manual annotations. Experimental results across multiple benchmarks demonstrate that EgoLoc significantly outperforms existing zero-shot methods, achieving a 100% success rate on the DeskTIL dataset and enabling high-performance robot policy learning.
Introduction
In the field of egocentric vision, understanding how humans interact with objects is a well-established area of research. However, understanding exactly when these interactions occur—the Temporal Interaction Localization (TIL) problem—remains a significant hurdle. Standard Temporal Action Localization (TAL) paradigms typically identify broad segments (e.g., "moving a bottle") but fail to pinpoint the specific frame where contact is first established or broken.
Existing methods often suffer from two primary issues: coarse granularity and perception gaps. Many models rely solely on 2D image data, leading to "scale aliasing" where a hand appears to touch an object due to 2D overlap, even if they are separated in 3D space. EgoLoc addresses these challenges by treating TIL as a fine-grained, zero-shot task that respects 3D physics.
Methodology
The EgoLoc pipeline is structured into three distinct stages designed to move from raw video data to precise interaction timestamps.
1. Self-Adaptive Sampling Strategy (SASS)
Rather than uniform sampling, EgoLoc utilizes 3D hand dynamics to identify likely interaction points. By extracting 3D wrist velocities and accelerations from RGB-D data, the system identifies "anchor frames" where velocity reaches a local minimum. These physical "lighthouses" serve as indicators of potential contact or separation events, allowing the model to ignore visual noise and focus on physically plausible moments.
2. VLM-Based Localization Module
EgoLoc employs a Vision-Language Model (VLM), such as GPT-4o, in a two-step localization process. First, a VLM Discriminator analyzes boundary frames to classify the event as "contact," "separation," or "neither." If an interaction is confirmed, a VLM Localizer examines a grid of candidate frames to determine the exact timestamp.
3. Closed-Loop Feedback Mechanism
To mitigate VLM hallucinations, EgoLoc introduces a VLM Checker. This module reviews the initial result; if an error is detected, it triggers a second round of reasoning. Crucially, it employs the failed first attempt as a negative in-context example, instructing the model to avoid its previous mistake. This feedback loop significantly reduces estimation uncertainty.
Experiments and Results
EgoLoc was evaluated on three benchmarks: EgoPAT3D-DT (re-annotated for fine-grained TIL), DeskTIL (featuring cluttered environments), and ManiTIL (long-form videos with sequential interactions).
Key findings include:
- High Precision: On the DeskTIL benchmark, EgoLoc achieved a Mean Absolute Error (MAE) of 0.611 seconds, vastly superior to the 2.333 seconds achieved by the leading baseline.
- Robustness: The system reached a 100% Success Rate (within a 5-frame tolerance) on the DeskTIL dataset.
- Robotic Application: Policies trained using EgoLoc-labeled data for a Franka Research 3 robot achieved an 80-100% success rate in manipulation tasks, outperforming traditional labeling methods.
Conclusion and Future Work
EgoLoc demonstrates that combining 3D physical priors with VLM-based reasoning provides a robust, zero-shot solution for fine-grained interaction localization. While highly effective, current limitations include a focus on single-hand interactions and decreased performance during rapid, non-impulsive actions. Future work will explore bimanual interactions and improving computational efficiency to reduce reliance on high-latency VLM APIs.
Summary updated