3D Hand Pose Estimation in Egocentric Images in the Wild

Aditya Prakash, Ruisen Tu, Matthew Chang, Saurabh Gupta

Abstract

We present WildHands, a method for 3D hand pose estimation in egocentric images in the wild. This is challenging due to (a) lack of 3D hand pose annotations for images in the wild, and (b) a form of perspective distortion-induced shape ambiguity that arises in the analysis of crops around hands. For the former, we use auxiliary supervision on in-the-wild data in the form of segmentation masks & grasp labels in addition to 3D supervision available in lab datasets. For the latter, we provide spatial cues about the location of the hand crop in the camera's field of view. Our approach achieves the best 3D hand pose on the ARCTIC leaderboard and outperforms FrankMocap, a popular and robust approach for estimating hand pose in the wild, by 45.3% when evaluated on 2D hand pose on our EPIC-HandKps dataset.

Introduction

Estimating 3D hand poses is a cornerstone for AR/VR and robotics, but doing it from a first-person perspective in everyday settings is notoriously difficult. Most existing models are trained in controlled laboratory settings with specialized sensors like MoCap or magnetic trackers. When these models move into "in-the-wild" environments—such as kitchens or workshops—they often fail due to poor visual signals (occlusions, motion blur), perspective distortion from wide-angle lenses, and a significant annotation gap where 3D labels for real-world egocentric video are virtually non-existent.

Methodology

WildHands addresses these challenges through a ResNet-50 based system that processes hand crops. Two key innovations drive its performance:

  1. Intrinsics-aware Positional Encoding (KPE): To mitigate perspective distortion, the model incorporates the hand crop's location relative to the camera's lens. By using the camera's focal length and principal point to calculate the angular location of the hand, the model can disambiguate shapes that appear distorted at the frame's edges.
  2. Auxiliary Supervision: To bridge the annotation gap, the authors employ a "mix-and-match" strategy. While lab datasets provide 3D ground truth, in-the-wild datasets (e.g., Epic-Kitchens) are supervised using 2D hand masks and grasp labels. Predicted 3D meshes are differentiably rendered into 2D masks for comparison, and a classifier ensures poses align with known grasp types.

The model iteratively refines MANO hand model parameters over three stages to produce the final 3D pose.

Experimental Evaluation

The study utilizes a diverse array of datasets, including lab-based sets (ARCTIC, AssemblyHands) and in-the-wild sets (Epic-Kitchens, Ego4D). To rigorously test 2D projection accuracy, the authors introduced Epic-HandKps, a benchmark containing 2D joint annotations for 5,000 images from Epic-Kitchens.

Results and Discussion

WildHands demonstrates significant improvements in zero-shot performance, outperforming robust baselines like FrankMocap by 7.4% to 66% across various metrics. Despite being 10x smaller and trained on 5x less data than the state-of-the-art transformer model HaMeR, WildHands remains highly competitive, beating HaMeR on 3 out of 6 metrics in system-level comparisons.

Qualitative comparison of WildHands against FrankMocap and HaMeR in challenging in-the-wild scenarios.

Key insights from the evaluation include:

  • Location Generalization: Using angular location (KPE) allows the model to generalize across different camera types, a critical requirement for egocentric vision.
  • Efficiency of Crops: Operating on hand crops rather than full images yielded a 27.7% gain in accuracy, as it allows the network to focus on fine-grained finger articulation.
  • Proxy Label Effectiveness: Training on 2D masks and grasp types provides sufficient constraint to learn realistic 3D structures without requiring explicit 3D joint labels for every frame.

Conclusion

WildHands provides a robust and efficient framework for 3D hand pose estimation in unconstrained egocentric environments. By leveraging camera intrinsics and auxiliary 2D supervision, it overcomes the primary hurdles of perspective distortion and data scarcity. Future work will focus on improving performance under extreme occlusion and automating the weighting of various supervision signals. This technology holds promise for scaling robot learning by enabling systems to accurately interpret human-object interactions from massive, unlabeled video libraries.

Summary updated

Abstract

We present WildHands, a method for 3D hand pose estimation in egocentric images in the wild. This is challenging due to (a) lack of 3D hand pose annotations for images in the wild, and (b) a form of perspective distortion-induced shape ambiguity that arises in the analysis of crops around hands. For the former, we use auxiliary supervision on in-the-wild data in the form of segmentation masks & grasp labels in addition to 3D supervision available in lab datasets. For the latter, we provide spatial cues about the location of the hand crop in the camera's field of view. Our approach achieves the best 3D hand pose on the ARCTIC leaderboard and outperforms FrankMocap, a popular and robust approach for estimating hand pose in the wild, by 45.3% when evaluated on 2D hand pose on our EPIC-HandKps dataset.
BibTeX
@inproceedings{prakash2024d,
  author = {Aditya Prakash and Ruisen Tu and Matthew Chang and Saurabh Gupta},
  title = {{3D Hand Pose Estimation in Egocentric Images in the Wild}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=uiwMRu3qDA},
}
Figure