TL;DR
This paper introduces WHSP-Net, a novel weakly-supervised deep learning approach for simultaneously estimating 3D hand shape and pose from a single depth image. It tackles the challenge of missing ground truth hand shape data in real-world datasets by cleverly leveraging synthetic data and a novel depth synthesizer for weak supervision, achieving state-of-the-art results.
What Problem Does This Paper Address?
Accurately reconstructing the 3D shape and pose of a human hand from a single image is crucial for many applications, including virtual and augmented reality, robotics, and animation. While significant progress has been made in 3D hand pose estimation, jointly recovering the detailed 3D hand shape (i.e., the mesh) alongside the pose remains a significant challenge. The primary hurdle is the lack of readily available ground truth data for hand shapes in real-world datasets. Manually annotating 3D hand meshes is extremely time-consuming and error-prone. Existing methods often rely on synthetic data or struggle with the complexity and variations in human hand shapes.
Methods
WHSP-Net is a three-component framework designed to overcome these limitations:
- Structured Hand Pose Estimation (Module 1): Instead of directly regressing 3D joint positions, this module first predicts 3D "bone vectors" (offsets between parent and child joints) using a CNN. A novel, parameter-free "bone-to-joint" layer then converts these bone vectors into 3D joint positions. This approach inherently respects the hand's kinematic structure, leading to more robust pose estimation.
- 3D Hand Shape Decoder (Module 2): This module takes the sparse 3D joint positions (output from Module 1) and decodes them into a dense 3D hand mesh. It's trained using a linear autoencoder-like architecture, learning to map pose to shape. Crucially, this module is trained using a mix of labeled synthetic data and unlabeled real data.
- 2D Depth Image Synthesizer (Module 3): This is the key to weak supervision. This module takes the reconstructed 3D hand mesh and synthesizes a 2D depth image from it. This synthesized depth image is then compared to the original input depth image, providing a supervisory signal for the hand shape estimation process. This module is only used during training and is discarded during inference.
The entire pipeline is trained end-to-end, allowing the modules to learn collaboratively.
Datasets
The paper utilizes a combination of datasets for training and evaluation:
- Synthetic Datasets:
- SynHand5M: A large synthetic dataset with 5 million depth images, providing ground truth for 21 3D joints and 1193 3D hand mesh vertices. This is crucial for supervised training of shape and pose components.
- Real-World Datasets:
- NYU: Contains 72,757 training and 8,252 test RGBD images. It provides ground truth for joint positions but not hand shape.
- BigHand2.2M: A large dataset with 956K training depth frames. It provides joint annotations but not ground truth for hand shape.
The authors also mention using a mixed training set combining synthetic and real data (e.g., $T_{SN} = T_S + T_V$ and $T_{BS} = T_B + T_S$).
Key Insights and Ideas
- Weak Supervision is Key: The lack of real-world hand shape ground truth is a major bottleneck. The paper demonstrates that a synthesized depth image can serve as an effective weak supervision signal, allowing the model to learn plausible hand shapes from unlabeled real data.
- Structure Matters for Pose: Directly regressing 3D joint positions can lead to geometrically invalid poses. By first predicting bone vectors and then converting them to joints, the model inherently enforces the hand's kinematic structure, improving pose accuracy.
- Pose as a Latent Representation for Shape: The paper leverages the idea that 3D joint positions can be seen as a sparse representation from which a dense hand mesh can be decoded. This forms the basis of their shape decoder.
- End-to-End Fine-tuning: Training all components together allows for better overall performance and adaptation between modules.
Key Results
- Synthetic Data Performance: WHSP-Net achieves state-of-the-art results on the SynHand5M dataset, outperforming previous methods that output more than just joint positions. Specifically, their "Full" model achieved a 3D V Err. of 5.12 mm, significantly better than DeepHPS (11.8 mm).
- Real-World Performance: The approach successfully reconstructs reasonable hand shapes from real-world datasets (NYU, BigHand2.2M) even with missing depth information and occlusions.
- Speed: The algorithm is real-time, producing both 3D hand mesh and pose in just 2.9 ms per frame.
- Comparison to Baselines: Self-comparison on the NYU dataset showed that incorporating hand structure (Baseline 2) improved pose accuracy by 9.6% over direct regression (Baseline 1). Adding shape learning (Baseline 3) further improved pose accuracy, and the full WHSP-Net (including the depth synthesizer) achieved the best results.
- Model 1 Comparison: A direct regression model (Model 1) that attempts to predict mesh vertices directly from depth images showed significantly worse pose estimation accuracy (23.63 mm 3D I Err. vs. 10.39 mm for WHSP-Net), highlighting the benefit of the structured approach and weak supervision.
Limitations and Future Work
- Dataset Consistency: The authors note that the quality of shape reconstruction depends on the consistency between real and synthetic image data. Datasets with inconsistent depth and joint annotations might pose challenges.
- Future Work: The authors plan to extend their approach by using 3D deep networks that establish a direct one-to-one relationship between voxelized depth images and the output 3D hand shape and pose representations.
Summary updated