EgoLoc: A Generalizable Solution for Temporal Interaction Localization in Egocentric Videos

Junyi Ma, Erhang Zhang, Yin-Dong Zheng, Yuchen Xie, Yixuan Zhou, Hesheng WangarXiv 2025 · DOI · arXiv

Abstract

Temporal Interaction Localization (TIL) in egocentric videos is critical for applications in robotics and AR/VR, yet remains challenging due to the need for fine-grained precision and 3D spatial awareness. We present EgoLoc, a zero-shot framework that localizes hand-object contact and separation moments. EgoLoc leverages 3D hand-motion dynamics for self-adaptive sampling and a closed-loop reasoning system powered by Vision-Language Models (VLMs). By incorporating physical priors and a feedback mechanism that utilizes negative in-context learning, EgoLoc achieves high-precision localization without manual annotations. Experimental results across multiple benchmarks demonstrate that EgoLoc significantly outperforms existing zero-shot methods, achieving a 100% success rate on the DeskTIL dataset and enabling high-performance robot policy learning.

Introduction

In the field of egocentric vision, understanding how humans interact with objects is a well-established area of research. However, understanding exactly when these interactions occur—the Temporal Interaction Localization (TIL) problem—remains a significant hurdle. Standard Temporal Action Localization (TAL) paradigms typically identify broad segments (e.g., "moving a bottle") but fail to pinpoint the specific frame where contact is first established or broken.

Comparison between the existing TAL paradigm and the proposed EgoLoc paradigm for temporal interaction localization.

Existing methods often suffer from two primary issues: coarse granularity and perception gaps. Many models rely solely on 2D image data, leading to "scale aliasing" where a hand appears to touch an object due to 2D overlap, even if they are separated in 3D space. EgoLoc addresses these challenges by treating TIL as a fine-grained, zero-shot task that respects 3D physics.

Methodology

The EgoLoc pipeline is structured into three distinct stages designed to move from raw video data to precise interaction timestamps.

1. Self-Adaptive Sampling Strategy (SASS)

Rather than uniform sampling, EgoLoc utilizes 3D hand dynamics to identify likely interaction points. By extracting 3D wrist velocities and accelerations from RGB-D data, the system identifies "anchor frames" where velocity reaches a local minimum. These physical "lighthouses" serve as indicators of potential contact or separation events, allowing the model to ignore visual noise and focus on physically plausible moments.

2. VLM-Based Localization Module

EgoLoc employs a Vision-Language Model (VLM), such as GPT-4o, in a two-step localization process. First, a VLM Discriminator analyzes boundary frames to classify the event as "contact," "separation," or "neither." If an interaction is confirmed, a VLM Localizer examines a grid of candidate frames to determine the exact timestamp.

The EgoLoc pipeline: (a) Self-Adaptive Sampling Strategy, (b) VLM-Based Localization Module, and (c) Closed-Loop Feedback Mechanism.

3. Closed-Loop Feedback Mechanism

To mitigate VLM hallucinations, EgoLoc introduces a VLM Checker. This module reviews the initial result; if an error is detected, it triggers a second round of reasoning. Crucially, it employs the failed first attempt as a negative in-context example, instructing the model to avoid its previous mistake. This feedback loop significantly reduces estimation uncertainty.

Experiments and Results

EgoLoc was evaluated on three benchmarks: EgoPAT3D-DT (re-annotated for fine-grained TIL), DeskTIL (featuring cluttered environments), and ManiTIL (long-form videos with sequential interactions).

Key findings include:

  • High Precision: On the DeskTIL benchmark, EgoLoc achieved a Mean Absolute Error (MAE) of 0.611 seconds, vastly superior to the 2.333 seconds achieved by the leading baseline.
  • Robustness: The system reached a 100% Success Rate (within a 5-frame tolerance) on the DeskTIL dataset.
  • Robotic Application: Policies trained using EgoLoc-labeled data for a Franka Research 3 robot achieved an 80-100% success rate in manipulation tasks, outperforming traditional labeling methods.

Conclusion and Future Work

EgoLoc demonstrates that combining 3D physical priors with VLM-based reasoning provides a robust, zero-shot solution for fine-grained interaction localization. While highly effective, current limitations include a focus on single-hand interactions and decreased performance during rapid, non-impulsive actions. Future work will explore bimanual interactions and improving computational efficiency to reduce reliance on high-latency VLM APIs.

Summary updated

BibTeX
@misc{ma2025egoloc,
  author = {Junyi Ma and Erhang Zhang and Yin-Dong Zheng and Yuchen Xie and Yixuan Zhou and Hesheng Wang},
  title = {{EgoLoc: A Generalizable Solution for Temporal Interaction Localization in Egocentric Videos}},
  year = {2025},
  eprint = {2508.12349},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.12349},
}
Figure