Egocentric human-object interaction detection: A new benchmark and method

Kunyuan Deng, Yi Wang, Lap-Pui Chau

TL;DR

This paper introduces Ego-HOIBench, the first large-scale, real-world egocentric dataset for human-object interaction (HOI) detection, addressing the critical gap in egocentric vision research. They also propose HGIR, a novel, lightweight method that leverages hand pose and geometric cues to significantly improve HOI detection in challenging egocentric scenarios, achieving state-of-the-art results.

What Problem Does This Paper Address?

Understanding how humans interact with objects is fundamental for intelligent agents, especially in egocentric (first-person) views. However, existing HOI detection methods, largely developed for third-person views, struggle with egocentric challenges like severe hand-object occlusion and limited context. Crucially, there's a lack of comprehensive egocentric HOI datasets with detailed annotations (hand, verb, object triplets). Existing egocentric datasets often focus on action recognition or lack fine-grained interaction details, hindering progress in this area.

Methods

The paper introduces the Hand Geometry and Interactivity Refinement (HGIR) scheme. HGIR is a plug-and-play module designed to enhance existing HOI detectors. It comprises four key components:

  • Hand Pose Estimation: Instead of relying on external pose estimators, HGIR integrates hand pose estimation directly into the HOI detection pipeline, sharing features with the hand detection branch. This allows for efficient extraction of hand pose candidates.
  • Hand Geometry Extraction: This component leverages the estimated hand poses to extract geometric features that capture the structural properties of hands. It uses a selection strategy to identify high-quality hand pose proposals and then computes geometric features based on joint angles and directional components.
  • Interactivity Refinement: This module uses a pose-interaction attention mechanism. It injects pose awareness into interaction features by using hand pose offset features as prompts, guiding the model to focus on relevant regions for interaction understanding.
  • Feature Aggregation: Finally, the global hand geometric features and the refined pose-aware interaction features are aggregated using an MLP to create a unified embedding that enhances interaction recognition and detection.

Datasets

The paper introduces Ego-HOIBench, a new benchmark dataset specifically for Ego-HOI detection.

  • Ego-HOIBench: This dataset contains over 27,575 RGB-D frames with explicit, fine-grained <human hand, verb, object> triplet annotations across 123 categories. It covers diverse daily scenarios, object types (rigid and articulated), and both single- and two-hand interactions. It addresses limitations of prior datasets by providing detailed annotations and a realistic testbed.

The authors also adapt and reimplement four representative third-person HOI detection methods on Ego-HOIBench for benchmarking.

Key Insights and Ideas

  • Egocentric HOI is Fundamentally Different: The paper strongly emphasizes that egocentric HOI presents unique challenges (occlusion, limited context) that necessitate specialized datasets and methods, unlike third-person HOI.
  • Hand Pose is Crucial for Egocentric HOI: Explicitly modeling hand pose and geometry is shown to be vital for robustly understanding interactions, especially under occlusion, where visual cues from objects might be obscured.
  • Integrated Hand Pose Estimation is Efficient: Embedding hand pose estimation within the HOI pipeline, rather than using external tools, significantly reduces computational overhead and allows for better adaptation to egocentric data.
  • Synergy of Geometric and Interaction Features: The HGIR scheme demonstrates that combining hand geometry features with refined interaction features leads to superior performance, highlighting the complementary nature of these cues.
  • Rare Interactions are a Major Challenge: The dataset reveals a significant imbalance in interaction frequencies, with rare triplets being particularly difficult to detect. The proposed method shows promise in improving the detection of these rare interactions.

Key Results

  • Ego-HOIBench Performance: Existing third-person HOI detectors show significant performance gaps when evaluated on Ego-HOIBench, underscoring the need for egocentric-specific solutions.
  • HGIR Improvements: Integrating HGIR with baseline HOI detectors (MUREN, QPIC) led to substantial performance gains. For example, with QPIC (ResNet-101 backbone), HGIR improved Full mAP by 1.7% and Top@G Accuracy by 6.2%.
  • State-of-the-Art Performance: The proposed method, using QPIC as a baseline with HGIR, achieved new state-of-the-art results on Ego-HOIBench, outperforming existing methods across various metrics, including rare triplet categories.
  • Robustness to Occlusion: HGIR consistently outperforms the baseline across different occlusion levels, showing significant improvements (e.g., 5.7% in Full mAP and 4.0% in Top@G Accuracy at high occlusion levels).
  • Efficiency: HGIR is lightweight and adds minimal computational overhead, with negligible drops in FPS and only a small increase in model parameters.

Limitations and Future Work

  • Imbalanced Data: The dataset exhibits a significant imbalance in the frequency of verb-object pairs, making it challenging to detect rare interactions.
  • Generalization to Rare Triplets: Future work will focus on addressing this imbalance and further improving model generalization to rare and complex Ego-HOI triplets.
  • Real-world Complexity: While Ego-HOIBench is a step towards real-world scenarios, further research is needed to handle even more complex and dynamic egocentric interactions.

Summary updated

Abstract

Egocentric human-object interaction (Ego-HOI) detection is crucial for intelligent agents to understand and assist human activities from a first-person perspective. However, progress has been hindered by the lack of benchmarks and methods tailored to egocentric challenges such as severe hand-object occlusion. In this paper, we introduce the real-world Ego-HOI detection task and the accompanying Ego-HOIBench, a new dataset with over 27K egocentric images and explicit, fine-grained hand-verb-object triplet annotations across 123 categories. Ego-HOIBench covers diverse daily scenarios, object types, and both single- and two-hand interactions, offering a comprehensive testbed for Ego-HOI research. Benchmarking existing third-person HOI detectors on Ego-HOIBench reveals significant performance gaps, highlighting the need for egocentric-specific solutions. To this end, we propose Hand Geometry and Interactivity Refinement (HGIR), a lightweight, plug-and-play scheme that leverages hand pose and geometric cues to enhance interaction representations. Specifically, HGIR explicitly extracts global hand geometric features from the estimated hand pose proposals, and further refines interaction features through pose-interaction attention, enabling the model to focus on subtle hand-object relationship differences even under severe occlusion. HGIR significantly improves Ego-HOI detection performance across multiple baselines, achieving new state-of-the-art results on Ego-HOIBench. Our dataset and method establish a solid foundation for future research in egocentric vision and human-object interaction understanding. Project page: https://dengkunyuan.github.io/EgoHOIBench/
BibTeX
@article{deng2026egocentric,
  author = {Kunyuan Deng and Yi Wang and Lap-Pui Chau},
  title = {{Egocentric human-object interaction detection: A new benchmark and method}},
  journal = {Expert Systems with Applications},
  year = {2026},
  url = {https://openreview.net/forum?id=FXs8qWjKXb},
}
Figure