Abstract
Precise localization of human-object interaction moments is a fundamental challenge in egocentric vision, essential for applications ranging from fine-grained behavior analysis to human-robot skill transfer. While traditional Temporal Action Localization (TAL) identifies broad action windows, it often fails to capture the exact millisecond of physical contact. We introduce EgoLoc, a zero-shot framework for Temporal Interaction Localization (TIL). EgoLoc combines 3D hand-motion analysis with the reasoning capabilities of Large Vision-Language Models (VLMs) like GPT-4o. By utilizing 3D velocity priors to narrow the search space and a closed-loop feedback mechanism for self-correction, EgoLoc achieves high-precision localization without task-specific training, significantly outperforming existing open-loop methods.
Introduction
Temporal Action Localization (TAL) typically focuses on identifying the start and end times of coarse actions. However, many real-world tasks require a more granular understanding of the "pre-contact," "in-contact," and "post-contact" phases of human-object interaction. This refined task, termed Temporal Interaction Localization (TIL), presents several challenges: existing models lack the precision for exact contact moments, require extensive task-specific annotations, and often ignore 3D motion cues that are critical for disambiguating physical interactions in 2D video.
EgoLoc addresses these limitations by shifting from an open-loop "one-shot" prediction to a closed-loop system that integrates physical motion priors with visual reasoning.
Methodology
The EgoLoc pipeline consists of three main stages designed to provide robust, zero-shot localization:
- Self-Adaptive Sampling: Leveraging RGB-D data, the system calculates 3D hand velocities. Based on the physical principle that hand speed reaches a local minimum during contact or separation, EgoLoc identifies "low-velocity" frames as high-probability anchor points. This significantly reduces the search space for the VLM.
- VLM-Based Reasoning: A "grid image" is constructed around the selected anchor frames and provided to a VLM (e.g., GPT-4o) alongside a universal text prompt. This allows the model to perform fine-grained visual reasoning within a pre-filtered temporal window.
- Closed-Loop Feedback: To mitigate VLM hallucinations, EgoLoc employs a dual-discriminator feedback loop. A visual discriminator (VLM-based) checks for obvious contact, while a dynamic discriminator (speed-based) ensures the hand velocity is consistent with a physical interaction. If the initial guess fails these checks, the system re-samples and re-reasons to refine the timestamp.
Evaluation and Results
EgoLoc was evaluated on the EgoPAT3D-DT dataset and a newly proposed benchmark, DeskTIL, which includes 150 sequences with varying levels of environmental clutter.
The framework demonstrated superior performance over the state-of-the-art zero-shot baseline, T-PIVOT:
- Accuracy: On DeskTIL, EgoLoc achieved a success rate of 0.929 (at $\gamma=3$), surpassing T-PIVOT's 0.908.
- Precision: The Mean Absolute Error (MAE) was reduced to 1.381 frames, compared to 1.767 frames for the best T-PIVOT configuration.
- Efficiency: By utilizing a more focused sampling strategy, EgoLoc achieved a 1.3x to 1.9x speedup over iterative reasoning methods.
Discussion and Limitations
A key insight of this work is that 3D motion priors effectively resolve depth ambiguities and scale aliasing inherent in 2D-only methods. Furthermore, the closed-loop mechanism provides a necessary safety net for VLM-based reasoning, which can otherwise be inconsistent.
Despite its strengths, EgoLoc currently assumes the interaction occurs in the first half of a video clip, limiting its direct application to long, untrimmed streams. Additionally, the reliance on depth data (RGB-D) restricts its use with standard RGB-only cameras. Future work will focus on integrating coarse TAL to identify interaction windows in long-form video before applying the high-precision EgoLoc pipeline.
Summary updated