Ego2HandsPose: A Dataset for Egocentric Two-Hand 3D Global Pose Estimation

Fanqing Lin, Tony MartinezWACV 2024

TL;DR

This paper introduces Ego2HandsPose, the first large-scale dataset for egocentric two-hand 3D global pose estimation in the wild, captured using a single RGB camera. They also present a novel annotation tool, ManoFit, that significantly simplifies the process of creating 3D hand pose annotations from 2D data, enabling the creation of this new dataset and the conversion of existing 2D datasets to 3D.

What Problem Does This Paper Address?

Accurate 3D hand pose estimation and tracking are crucial for immersive applications like VR/AR, especially with the rise of the metaverse. While single-hand pose estimation has seen progress, two-hand estimation, particularly in unconstrained, "in-the-wild" environments using only a standard RGB camera, remains a significant challenge. Existing datasets are often limited to laboratory settings, third-person viewpoints, or rely on expensive depth sensors, hindering the development of robust models that can generalize to real-world scenarios. The complexity of annotating 3D hand poses, especially for two hands, has been a major bottleneck in creating diverse and large-scale datasets.

Methods

The core of this work lies in two main contributions:

  • ManoFit Annotation Tool: The authors developed a parametric fitting algorithm called ManoFit. This tool leverages the deformable MANO hand model and allows users to annotate 3D hand poses using a single RGB image with minimal manual guidance. It works by fitting the MANO model to 2D keypoints, incorporating physical constraints for natural poses, and using a multi-stage optimization process to refine global orientation, pose articulation, and global translation. This significantly simplifies the annotation process compared to traditional multi-view or depth-based methods.
  • Ego2HandsPose Dataset Creation: Using ManoFit, the authors annotated approximately 7,000 frames from the existing Ego2Hands dataset (which focused on segmentation/detection) to create Ego2HandsPose. They also applied ManoFit to convert existing 2D hand pose datasets (HIU-DMTL, PanHand2D, OneHand10k) into 3D datasets, creating a synthetic dataset called MANO3Dhands for training.

The paper also outlines a multi-stage pipeline for two-hand 3D global pose estimation, which includes:

  1. Two-hand segmentation and detection: Using an adapted ICNet.
  2. 2D hand pose estimation: Employing HRNet-W32 trained on their composited training data.
  3. 3D canonical hand pose estimation: Using a compact ResNet10 trained on MANO3Dhands.
  4. Hand tracking via ManoFit: Applying the ManoFit algorithm for temporally consistent tracking.

Datasets

  • Ego2HandsPose (New Dataset): The primary contribution, containing 3D hand pose annotations for egocentric, two-hand scenarios captured in the wild using a single RGB camera. It's an extension of the Ego2Hands dataset.
  • Ego2Hands (Existing Dataset): Used as a source for creating Ego2HandsPose. It provided segmentation and detection data.
  • HIU-DMTL, PanHand2D, OneHand10k (Existing Datasets): These 2D hand pose datasets were converted to 3D using the ManoFit tool.
  • MANO3Dhands (New Synthetic Dataset): A synthetic dataset generated using ManoFit and real-world pose distributions, designed to provide a diverse pose space for training 3D canonical pose estimation models.
  • FreiHAND (Existing Dataset): Used for quantitative evaluation of the ManoFit algorithm's accuracy.
  • H2O, FreiHAND (Existing Datasets): Used for evaluating the complete two-hand 3D global pose estimation pipeline.

Key Insights and Ideas

  • Simplifying 3D Hand Pose Annotation: The ManoFit tool demonstrates that high-quality 3D hand pose annotations can be generated from single RGB images, significantly reducing the annotation burden and enabling the creation of larger, more diverse datasets.
  • Egocentric Viewpoint is Crucial: The paper strongly advocates for egocentric datasets for applications like VR/AR, as third-person viewpoints impose constraints on user interaction and are less representative of the target use cases.
  • Domain Gap in Pose Space: Different viewpoints (egocentric vs. third-person) and environments (synthetic vs. real-world) lead to distinct data distributions, highlighting the importance of training on data that matches the target domain for optimal generalization.
  • Compositional Data Generation: The Ego2Hands dataset's approach of compositing two-hand instances at training time is effective for generating diverse data and improving generalization, a principle extended in this work.

Key Results

  • 2D Hand Pose Estimation: Training on Ego2HandsPose significantly outperforms other datasets, achieving the top AUC2D of 0.536 on their test set.
  • 3D Canonical Hand Pose Estimation: The proposed MANO3Dhands dataset shows the best generalization scores across various datasets, indicating its effectiveness in representing diverse pose spaces.
  • Two-hand 3D Global Pose Estimation: The complete pipeline trained on Ego2HandsPose achieves state-of-the-art results, outperforming models trained on existing datasets like H2O and FreiHAND by a significant margin (e.g., AUCangle of 0.912 vs. 0.862 for H2O).
  • ManoFit Accuracy: The ManoFit algorithm achieves an End Point Error (EPE) of 1.17cm when evaluated on FreiHAND.
  • Acceptance Rates: High acceptance rates (e.g., 84.6% for HIU-DMTL) for the converted 3D datasets indicate the robustness of the ManoFit algorithm.

Limitations and Future Work

  • Real-time Performance: The current pipeline does not run in real-time, with average inference times of 19.7ms, 50.1ms, and 2.4ms for segmentation, 2D pose, and 3D pose estimation respectively. Future work will focus on optimizing efficiency.
  • Subject-Specific Shape: While the ManoFit tool achieves good fitting accuracy without subject-specific shape fitting, it's acknowledged that incorporating this could further improve results.
  • Limited Scope of Existing Datasets: The paper repeatedly emphasizes the limitations of existing datasets in terms of diversity, viewpoint, and real-world applicability, motivating the creation of Ego2HandsPose.
  • Future Work: The authors hope their work will push color-based two-hand applications towards unconstrained environments, implying continued research in this direction. Further optimization for real-time performance is also a clear future direction.

Summary updated

BibTeX
@inproceedings{lin2024ego,
  author = {Fanqing Lin and Tony Martinez},
  title = {{Ego2HandsPose: A Dataset for Egocentric Two-Hand 3D Global Pose Estimation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure