3D Hand Pose Estimation in Everyday Egocentric Images

Aditya Prakash, Ruisen Tu, Matthew Chang, Saurabh Gupta

TL;DR

WildHands is a new system for 3D hand pose estimation designed specifically for the challenges of "in-the-wild" egocentric (first-person) video. By combining intrinsics-aware positional encodings to handle perspective distortion with auxiliary 2D supervision from massive unlabelled datasets, it achieves state-of-the-art performance while being 10x smaller and using 5x less training data than leading transformer-based models.

What Problem Does This Paper Address?

Estimating the 3D pose of a hand from a single first-person image is notoriously difficult. Unlike "exocentric" views (where a camera looks at a person from a distance), egocentric views suffer from:

  1. Severe Perspective Distortion: Hands are very close to the lens, meaning small movements in 3D result in large, non-linear changes in the 2D image.
  2. Heavy Occlusion: Hands are almost always interacting with objects, hiding fingers and joints.
  3. Data Scarcity: While we have high-quality 3D annotations from controlled lab settings (using MoCap or magnetic sensors), we lack ground-truth 3D labels for everyday activities like cooking or cleaning in real homes.

Existing methods often use image crops to focus on the hand, but this discards crucial information about where the hand is relative to the camera, leading to "shape ambiguity"—where different 3D poses look identical in a 2D crop.

Methods

The authors introduce WildHands, a system that systematically addresses these gaps through three main pillars:

  • Intrinsics-aware Positional Encoding (KPE): To solve the ambiguity caused by cropping, the model doesn't just look at the hand pixels. It uses the camera's focal length and principal point to calculate the angular location of each pixel in the camera's field of view. This "KPE" embedding is concatenated with the image features, allowing the network to understand the perspective geometry of the crop.
  • Auxiliary Supervision: To train on "in-the-wild" data without 3D labels, the authors use two clever proxies:
  • 2D Masks: They use a differentiable renderer to project the predicted 3D hand mesh back into 2D and compare it against 2D segmentation masks.
  • Grasp Classification: Since hand pose is highly correlated with how we grip objects, they train a small head to predict the "grasp type" (e.g., power grip, pinch) and use that as an additional training signal.
  • Architecture: The system uses a standard ResNet50 backbone and a MANO-based decoder (a parametric model of the human hand), making it significantly more efficient than massive Vision Transformer (ViT) alternatives.
The WildHands architecture uses KPE to inform the encoder and leverages auxiliary losses like masks and grasps for training on in-the-wild data.

Datasets

The study utilizes a mix of lab-controlled and real-world datasets:

  • Training (Lab): ARCTIC and AssemblyHands (provide 3D ground truth).
  • Training (In-the-wild): Epic-Kitchens and Ego4D (provide 2D masks and grasp labels).
  • Evaluation: H2O, AssemblyHands, and Ego-Exo4D.
  • New Benchmark: The authors introduce Epic-HandKps, a new evaluation set containing 5,000 images from Epic-Kitchens with manual 2D joint annotations to test how well models generalize to everyday environments.
Examples of the new Epic-HandKps dataset, showing the high degree of occlusion and dexterous poses found in real-world kitchen tasks.

Key Insights and Ideas

  • Crops are a double-edged sword: While cropping helps the model see fine details, it breaks the perspective geometry. KPE is the "missing link" that restores this geometric context.
  • Absolute vs. Relative Pose: KPE is particularly effective at improving absolute pose (where the hand is in space) rather than just the relative articulation of the fingers.
  • Masks > Grasps: For auxiliary supervision, 2D segmentation masks provide a much stronger learning signal than grasp labels, though using both is best.
  • Efficiency Wins: A well-designed convolutional model with proper geometric priors can outperform or match massive transformers (like HaMeR) while being an order of magnitude smaller.

Key Results

WildHands demonstrates significant improvements across the board:

  • Zero-shot Generalization: On four diverse datasets, it outperformed previous methods by 7.4% to 66% in 2D and 3D metrics.
  • ARCTIC Leaderboard: As of July 2024, WildHands holds the #1 spot on the egocentric split of the ARCTIC leaderboard, beating both convolutional and transformer-based competitors.
  • System Comparison: Compared to HaMeR (a state-of-the-art transformer model), WildHands is 10x smaller and trained on 5x less data, yet it wins on 3 out of 6 key metrics.
Qualitative comparisons show WildHands (bottom rows) correctly capturing finger articulation during object interaction where other models like FrankMocap or HaMeR fail.

Limitations and Future Work

  • Intrinsics Dependency: The KPE method requires knowing the camera's internal parameters (focal length, etc.). While often available in metadata, this isn't always the case for random internet videos.
  • Loss Weighting: The model currently uses fixed hyperparameters to balance different loss terms (3D vs. 2D vs. Grasp). The authors suggest that automated, uncertainty-based loss weighting could further improve performance.
  • Extreme Cases: The model still struggles with "extreme" poses or scenarios where fingers are almost entirely hidden, such as kneading dough.
Fig. 1: WildHands predicts the 3D shape, 3D articulation and 3D placement of the hand in the camera frame from a single in-the-wild egocentric RGB image and camera intrinsics. It produces better 3D output compared to FrankMocap [59] in occlusion scenarios and is more adept at dealing with perspective distortion than HaMeR [52], in challenging egocentric hand-object interactions from Epic-Kitchens [9] dataset.

Summary updated

Abstract

3D hand pose estimation in everyday egocentric images is challenging for several reasons: poor visual signal (occlusion from the object of interaction, low resolution & motion blur), large perspective distortion (hands are close to the camera), and lack of 3D annotations outside of controlled settings. While existing methods often use hand crops as input to focus on fine-grained visual information to deal with poor visual signal, the challenges arising from perspective distortion and lack of 3D annotations in the wild have not been systematically studied. We focus on this gap and explore the impact of different practices, i.e. crops as input, incorporating camera information, auxiliary supervision, scaling up datasets. We provide several insights that are applicable to both convolutional and transformer models leading to better performance. Based on our findings, we also present WildHands, a system for 3D hand pose estimation in everyday egocentric images. Zero-shot evaluation on 4 diverse datasets (H2O, AssemblyHands, Epic-Kitchens, Ego-Exo4D) demonstrate the effectiveness of our approach across 2D and 3D metrics, where we beat past methods by 7.4% - 66%. In system level comparisons, WildHands achieves the best 3D hand pose on ARCTIC egocentric split, outperforms FrankMocap across all metrics and HaMeR on 3 out of 6 metrics while being 10x smaller and trained on 5x less data.
BibTeX
@inproceedings{prakash2024d,
  author = {Aditya Prakash and Ruisen Tu and Matthew Chang and Saurabh Gupta},
  title = {{3D Hand Pose Estimation in Everyday Egocentric Images}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=uiwMRu3qDA},
}
Figure