Zero-Shot Temporal Interaction Localization for Egocentric Videos

Wang Hesheng, Erhang Zhang, Junyi Ma, Yin-Dong Zheng, Yixuan Zhou, Fan XuIROS 2025

Abstract

Precise localization of human-object interaction moments is a fundamental challenge in egocentric vision, essential for applications ranging from fine-grained behavior analysis to human-robot skill transfer. While traditional Temporal Action Localization (TAL) identifies broad action windows, it often fails to capture the exact millisecond of physical contact. We introduce EgoLoc, a zero-shot framework for Temporal Interaction Localization (TIL). EgoLoc combines 3D hand-motion analysis with the reasoning capabilities of Large Vision-Language Models (VLMs) like GPT-4o. By utilizing 3D velocity priors to narrow the search space and a closed-loop feedback mechanism for self-correction, EgoLoc achieves high-precision localization without task-specific training, significantly outperforming existing open-loop methods.

Introduction

Temporal Action Localization (TAL) typically focuses on identifying the start and end times of coarse actions. However, many real-world tasks require a more granular understanding of the "pre-contact," "in-contact," and "post-contact" phases of human-object interaction. This refined task, termed Temporal Interaction Localization (TIL), presents several challenges: existing models lack the precision for exact contact moments, require extensive task-specific annotations, and often ignore 3D motion cues that are critical for disambiguating physical interactions in 2D video.

Comparison between traditional TAL (open-loop) and the proposed EgoLoc paradigm (closed-loop) for TIL.

EgoLoc addresses these limitations by shifting from an open-loop "one-shot" prediction to a closed-loop system that integrates physical motion priors with visual reasoning.

Methodology

The EgoLoc pipeline consists of three main stages designed to provide robust, zero-shot localization:

  1. Self-Adaptive Sampling: Leveraging RGB-D data, the system calculates 3D hand velocities. Based on the physical principle that hand speed reaches a local minimum during contact or separation, EgoLoc identifies "low-velocity" frames as high-probability anchor points. This significantly reduces the search space for the VLM.
  2. VLM-Based Reasoning: A "grid image" is constructed around the selected anchor frames and provided to a VLM (e.g., GPT-4o) alongside a universal text prompt. This allows the model to perform fine-grained visual reasoning within a pre-filtered temporal window.
  3. Closed-Loop Feedback: To mitigate VLM hallucinations, EgoLoc employs a dual-discriminator feedback loop. A visual discriminator (VLM-based) checks for obvious contact, while a dynamic discriminator (speed-based) ensures the hand velocity is consistent with a physical interaction. If the initial guess fails these checks, the system re-samples and re-reasons to refine the timestamp.
The EgoLoc pipeline: (a) Self-Adaptive Sampling using 3D motion, (b) VLM-Based Reasoning with grid images, and (c) the Closed-Loop Feedback Mechanism.

Evaluation and Results

EgoLoc was evaluated on the EgoPAT3D-DT dataset and a newly proposed benchmark, DeskTIL, which includes 150 sequences with varying levels of environmental clutter.

The framework demonstrated superior performance over the state-of-the-art zero-shot baseline, T-PIVOT:

  • Accuracy: On DeskTIL, EgoLoc achieved a success rate of 0.929 (at $\gamma=3$), surpassing T-PIVOT's 0.908.
  • Precision: The Mean Absolute Error (MAE) was reduced to 1.381 frames, compared to 1.767 frames for the best T-PIVOT configuration.
  • Efficiency: By utilizing a more focused sampling strategy, EgoLoc achieved a 1.3x to 1.9x speedup over iterative reasoning methods.

Discussion and Limitations

A key insight of this work is that 3D motion priors effectively resolve depth ambiguities and scale aliasing inherent in 2D-only methods. Furthermore, the closed-loop mechanism provides a necessary safety net for VLM-based reasoning, which can otherwise be inconsistent.

Despite its strengths, EgoLoc currently assumes the interaction occurs in the first half of a video clip, limiting its direct application to long, untrimmed streams. Additionally, the reliance on depth data (RGB-D) restricts its use with standard RGB-only cameras. Future work will focus on integrating coarse TAL to identify interaction windows in long-form video before applying the high-precision EgoLoc pipeline.

Summary updated

BibTeX
@inproceedings{hesheng2025zero,
  author = {Wang Hesheng and Erhang Zhang and Junyi Ma and Yin-Dong Zheng and Yixuan Zhou and Fan Xu},
  title = {{Zero-Shot Temporal Interaction Localization for Egocentric Videos}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
}
Figure