Abstract
Recently, there has been an increase in research estimating hand poses using images. Due to the hand's high degree of freedom and self-occlusion, multi-view or depth images are often used. Our objective was to estimate hand poses specifically while grasping objects. When holding something, the hand moves in many directions. However, if the camera is too distant from the hand, it may move out of range. Widening the viewing angle, however, reduces the resolution beyond usable limits. One possible solution was developed by Kashiwagi -by setting the camera on an object, the hand's pose can be estimated regardless of its position. However, Kashiwagi's method cannot be used without estimating the fingertips' positions. Recently, another method using a convolutional neural network (CNN), useful for estimating complex poses, has been proposed. Unfortunately, it is difficult to collect the large number of images with ground truth needed for learning. In this research, we focused on creating a large dataset by generating hand pose images using a digital human model and motioncaptured data using DhaibaWorks. We evaluated the model by calculating the distance of the estimated pose and ground truth of the test data, which was approximately 12.3 mm on average. In comparison, the average distance in related work was 18.5 mm. We also tested our method with ordinary camera images and confirmed that it can be used in the real world. Our method provides a new means of dataset generation: annotations are done automatically with motion capture technology, which reduces the time required. In future work, we will improve the architecture of the CNN and shorten the execution time for real-time processing.
Introduction
Estimating the 3D pose of a human hand is a fundamental challenge in computer vision, complicated by the hand's high degrees of freedom and frequent self-occlusion. These challenges are exacerbated during object grasping, where the object itself obstructs the view of the fingers. While depth sensors or multi-camera arrays offer solutions, they are often impractical for consumer-grade applications. Consequently, there is a strong push toward using standard RGB images. However, training robust deep learning models requires massive datasets with precise 3D annotations—a task that is prohibitively expensive and error-prone when performed manually.
Methodology
This research introduces a novel pipeline that automates the generation of high-quality training data using digital human models and motion capture technology.
The process begins with Motion Capture, where 28 physical markers are placed on a participant's hand to record grasping motions (e.g., holding a transparent sphere) using a Vicon Nexus system.
This captured data is integrated into DhaibaWorks, a digital human modeling software, to generate a realistic 3D CG hand model. By manipulating this model, the researchers can automatically generate thousands of synthetic images from various camera angles. To enhance model robustness, the pipeline includes alterations to skin color and the addition of background noise, all while maintaining perfect "ground truth" 3D coordinates for every joint.
Finally, a Convolutional Neural Network (CNN) based on the Inception V4 architecture is trained on this synthetic data. The network maps a single RGB input image to 84 output values, representing the (x, y, z) coordinates for 28 keypoints.
Datasets and Evaluation
The study utilized two primary synthetic datasets created via the DhaibaWorks pipeline:
- Dataset A: 5,810 images with complex, "noised" backgrounds to simulate real-world clutter.
- Dataset B: 5,810 images with clean backgrounds.
- Evaluation Set: 113 images for accuracy testing.
Additionally, the model was tested on real-world photographs to verify its ability to generalize from synthetic CG hands to actual human hands.
Results and Discussion
The proposed method achieved a mean estimation error of 12.31 mm, significantly outperforming the 18.5 mm error reported in related work.
Key insights from the evaluation include:
- Synthetic-to-Real Generalization: The model successfully estimated poses from real photos, proving that synthetic data generated via digital human models is a viable substitute for manually labeled real-world data.
- Impact of Background Noise: Dataset A (noisy background) yielded slightly better performance than Dataset B, suggesting that background complexity encourages the CNN to learn more discriminative hand features.
- Joint Sensitivity: Accuracy was highest for the index and middle fingers, while the little finger tip—often the most occluded during a grasp—showed slightly higher error rates.
- Viewpoint Optimization: Adjusting the virtual camera closer to the hand further reduced the error to 11.31 mm.
Conclusion
This research demonstrates that combining digital human modeling with CNNs can effectively solve the data scarcity problem in 3D hand pose estimation. While the accuracy is high, the current Inception V4 implementation requires approximately 7.41 seconds per image, which is unsuitable for real-time use. Future work will focus on optimizing the CNN architecture for speed, expanding the variety of grasped objects, and implementing region-of-interest pre-processing to improve overall efficiency.
Summary updated