TL;DR
This paper introduces the first method for estimating the full 3D global pose of both hands from a single, standard RGB image, a significant step towards more natural human-computer interaction. They achieve this by developing a novel multi-stage pipeline and a new synthetic dataset to overcome the challenges of occlusion and lack of depth information.
What Problem Does This Paper Address?
Accurate hand pose estimation is crucial for applications like gesture control, virtual reality, and human-computer interaction. While significant progress has been made, most existing methods rely on depth cameras or multiple RGB cameras, which are costly and limit applicability. Even for single-hand 3D pose estimation using only RGB, challenges like complex backgrounds and occlusions persist. However, estimating the global 3D pose of two hands simultaneously from a single RGB image is particularly difficult due to inter-hand occlusions and the inherent ambiguity of depth estimation from 2D images. This paper tackles this challenging problem, aiming to provide a more accessible and versatile solution.
Methods
The authors propose a novel four-component pipeline:
- Hand Segmentation and Detection (HandSegNet): A residual network is used to simultaneously segment the hands and generate heatmap energies that highlight hand locations, even when occluded. This information is crucial for accurately separating and processing each hand.
- 2D Hand Pose Estimation (PoseNet2D): A variant of Convolutional Pose Machines (CPM) is employed to estimate 2D joint locations. To improve performance, they introduce "z-heatmaps" which encode depth information into the 2D heatmaps, and they leverage the segmentation masks from HandSegNet to mitigate inter-hand occlusion.
- 3D Canonical Hand Pose Estimation (PoseNet3D): This component lifts the 2D heatmaps to estimate 3D canonical joint locations for each hand. The canonical frame is defined such that the middle metacarpophalangeal (mMCP) joint is at the origin, and the distance between the wrist and mMCP is normalized. They use a loss function that enforces consistency between 2D and 3D poses and bone lengths.
- 3D Global Hand Pose Estimation: This is the core novelty. They introduce a new algorithm that uses the estimated 2D and 3D canonical poses, along with the actual length of a key bone and camera intrinsics, to reconstruct the absolute 3D global joint locations in Cartesian coordinates. This involves a clever projection algorithm that accounts for depth ambiguity by using a spherical coordinate system.
Datasets
- Ego3DHands Dataset: The authors introduce a new, large-scale synthetic dataset for training and evaluating two-hand global 3D pose estimation. It includes both static and dynamic hand poses captured from an egocentric viewpoint, generated using Blender. This dataset is designed to address the lack of real-world annotated data for this specific task.
- Stereo Tracking Benchmark Dataset (STB): Used for evaluating single-hand 3D canonical pose estimation.
- Rendered Hand Pose Dataset (RHP): Also used for evaluating single-hand 3D canonical pose estimation.
Key Insights and Ideas
- Segmentation is Key for Occlusion: Accurately segmenting hands is vital for resolving inter-hand occlusions and improving 2D pose estimation, especially when hands have similar appearances.
- "Z-Heatmaps" Encode Depth: By generating stronger heatmap energy for closer joints, they implicitly encode depth information into the 2D heatmaps, which aids in 3D pose estimation.
- Spherical Coordinates for Global Pose: The inherent depth ambiguity in monocular RGB makes direct Cartesian estimation challenging. Using spherical coordinates for the global pose estimation provides a more robust way to handle rotations and distances.
- Synthetic Data Bridges the Gap: The creation of the Ego3DHands dataset is crucial for training robust models for this novel task, overcoming the limitations of real-world data availability and annotation costs.
Key Results
- State-of-the-Art Single-Hand Performance: Their PoseNet2D and PoseNet3D models achieve competitive or superior results on single-hand 3D canonical pose estimation benchmarks (STB and RHP) compared to existing methods, even when using only RGB input. For example, on STB, they achieve an AUC of 0.995.
- Promising Two-Hand Global Pose Estimation: On their newly introduced Ego3DHands dataset, their method demonstrates promising results for two-hand global 3D pose estimation, achieving directional and distance accuracy of approximately 0.90 at an angular threshold of 3 degrees and a radius threshold of 7 cm.
- Robustness to Real-World Data: Qualitative results on real-world datasets (STB and RHP) show that their global estimation algorithm generalizes well, even with limited training data.
Limitations and Future Work
- Real-World Data Annotation: The authors acknowledge that evaluating their method on real-world data requires sufficiently annotated training data, which is currently a bottleneck.
- Improvement in Distance Accuracy: While promising, the authors note that there is "definitely room for improvement" in the distance accuracy for global pose estimation, as hand poses with a 7 cm difference in distance can appear visually similar in 2D images.
- Future Work: Future work will focus on collecting and annotating more real-world data to further improve performance and extend the evaluation to other datasets and scenarios.
Summary updated