Abstract
Estimating 3D hand shape and pose from a single RGB image is a complex task due to high dimensionality and self-occlusions. This paper demonstrates that self-supervised pretraining on domain-specific hand data significantly outperforms standard ImageNet pretraining. By introducing HanCo, a large-scale multi-view dataset, and leveraging advanced contrastive sampling strategies, the authors achieve a 4.7% reduction in mesh error and a 3.6% boost in F-score.
Problem Statement
Acquiring high-quality 3D labels for hand meshes is expensive and time-consuming. Traditionally, researchers use transfer learning from ImageNet-pretrained models. However, ImageNet features are often suboptimal for the geometric nuances of hand articulation. This work explores whether unlabeled hand images can learn a better starting representation than generic object datasets.
Methodology
The authors propose a two-stage training approach:
- Self-Supervised Pretraining: Utilizing Momentum Contrastive (MoCo) learning, an encoder is trained to produce similar embeddings for different views of the same hand pose while separating different poses.
- Supervised Fine-tuning: The pretrained weights initialize a network trained on a smaller labeled dataset to predict MANO parameters for 3D hand modeling.
Advanced Sampling Strategies
To maximize the effectiveness of contrastive learning, the authors exploit the multi-view nature of their dataset:
- Multi-view Sampling: Synchronized cameras provide images of the same pose from different angles, forcing the model to learn viewpoint-invariant features.
- Background Randomization: Hands are "cut and pasted" onto random backgrounds to ensure the model focuses on hand geometry rather than the environment.
- Temporal Sampling: Neighboring video frames are used to enforce representation consistency across slight movements.
The HanCo Dataset
The HanCo (Hand Contrastive) dataset consists of 107,538 unique poses recorded by 8 synchronized cameras, totaling over 860,000 frames. It includes multi-view data, temporal sequences, and green-screen setups. For evaluation, the authors use FreiHAND, a subset with high-quality annotations, and a collection of 2,193 Flickr backgrounds for randomization.
Key Insights
- Domain-Specific Pretraining: Pretraining on hands alone provides more relevant features than general-purpose datasets like ImageNet.
- The Power of Multi-view: Seeing the same pose from multiple angles is a more powerful augmentation than standard image manipulations, directly targeting the geometric nature of the task.
- Label Efficiency: Self-supervised pretraining is most beneficial with moderate amounts of labeled data (approx. 40%), where it provides a strong prior that labeled data alone might miss.
Key Results
The proposed Ours-Multi view model reduced 3D mesh error to 1.02 cm (down from 1.07 cm for the ImageNet baseline) and improved the F-score at a 5mm threshold to 0.548.
Qualitatively, the model captures global pose and complex "grasps" more accurately, particularly in orientations where standard models typically struggle.
Limitations and Future Work
Current sampling strategies only scratch the surface; more complex temporal consistency could further improve results. Future work includes combining this pretraining with weak supervision (e.g., 2D keypoints) and testing on "in-the-wild" footage with varied lighting and motion blur, as HanCo was recorded in controlled settings.
Summary updated