Interacting Two-Hand 3D Pose and Shape Reconstruction from Single Color Image

Baowen Zhang, Yangang Wang, Xiaoming Deng, Yinda Zhang, Ping Tan, Cuixia Ma, Hongan Wang

Abstract

Reconstructing the 3D pose and shape of two interacting hands from a single RGB image is a complex task due to significant inter-hand occlusion and visual ambiguity. We present a novel deep learning framework that leverages a pose-aware attention mechanism and a context-aware cascaded refinement process. By treating the hands as a correlated pair and utilizing explicit heatmap-guided attention, our model achieves state-of-the-art performance on the InterHand2.6M and CMU Haggling datasets, significantly reducing joint position error.

Introduction

While 3D reconstruction of a single hand has matured, the interaction between two hands presents unique challenges. Heavy self-occlusion and inter-hand occlusion—where one hand partially or fully hides the other—are common. Furthermore, the near-identical texture and color of both hands often cause neural networks to misassign features, leading to "hand confusion." Existing high-fidelity solutions typically require depth sensors or multi-view setups. This research aims to achieve accurate 3D mesh reconstruction of both hands using only a standard, single-view color camera.

Methodology

The proposed framework moves away from independent hand processing, instead modeling the hands as a correlated pair.

Interacting Hand Representation

The system employs the MANO parametric model to describe hand shape ($\beta$) and pose ($\theta$). It also predicts relative translation and scale to accurately position both hands in 3D space.

Pose-aware Feature Extraction and Refinement

To mitigate feature confusion, the network first predicts 2.5D heatmaps for the joints. These heatmaps generate an attention mask that guides the network to extract distinct features for each hand, even during overlap.

Overview of the proposed multi-stage network architecture featuring 2.5D heatmap prediction and cascaded refinement.

The initial 3D estimate undergoes a Context-aware Cascaded Refinement. This stage exploits the physical constraints inherent in hand interactions, iteratively refining the shape and pose of both hands simultaneously. A Shape Consistency Loss is also applied, ensuring that both hands (belonging to the same subject) maintain consistent shape parameters.

Key Insights

A core finding of this work is the high degree of correlation between the poses of interacting hands. Visual analysis using t-SNE confirms that while unpaired hands appear random, interacting hands follow specific, predictable patterns. The "context-aware" module leverages this mathematical correlation to infer the positions of occluded joints. Additionally, explicitly guiding attention via heatmaps, rather than relying on black-box learning, proves crucial for maintaining feature identity between the left and right hands.

Experimental Results

The model was evaluated on the InterHand2.6M and CMU Haggling datasets, outperforming existing state-of-the-art (SOTA) methods.

Qualitative reconstruction results demonstrating the model's ability to handle complex, intertwined hand poses from various viewpoints.

On the InterHand2.6M benchmark, the framework achieved a Mean Per Joint Position Error (MPJPE) of 13.07mm, a notable improvement over previous benchmarks (e.g., 15.40mm by Spurr et al.). The vertex-to-vertex shape error was reduced to 10.39mm, nearly halving the error rates of single-hand methods adapted for two-hand tasks. Qualitative results demonstrate the model's robustness in reconstructing complex "intertwined" poses without the mesh penetration or "broken" geometry seen in prior work.

Conclusion and Future Work

This framework establishes a new standard for single-image two-hand reconstruction by emphasizing inter-hand correlation and guided attention. Future improvements could include integrating detection directly into the pipeline to remove bounding box dependencies, implementing physics-based losses to prevent mesh interpenetration, and extending the method to video sequences for improved temporal consistency in real-time applications.

Summary updated

Abstract

In this paper, we propose a novel deep learning framework to reconstruct 3D hand poses and shapes of two interacting hands from a single color image. Previous methods designed for single hand cannot be easily applied for the two hand scenario because of the heavy inter-hand occlusion and larger solution space. In order to address the occlusion and similar appearance between hands that may confuse the network, we design a hand pose-aware attention module to extract features associated to each individual hand respectively. We then leverage the two hand context presented in interaction to propose a context-aware cascaded refinement that improves the hand pose and shape accuracy of each hand conditioned on the context between interacting hands. Extensive experiments on the main benchmark datasets demonstrate that our method predicts accurate 3D hand pose and shape from single color image, and achieves the state-of-the-art performance. Code is available in project webpage https://baowenz.github.io/Intershape/.
BibTeX
@inproceedings{zhang2021interacting,
  author = {Baowen Zhang and Yangang Wang and Xiaoming Deng and Yinda Zhang and Ping Tan and Cuixia Ma and Hongan Wang},
  title = {{Interacting Two-Hand 3D Pose and Shape Reconstruction from Single Color Image}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01116},
}
Figure