Monocular 3D Reconstruction of Interacting Hands via Collision-Aware Factorized Refinements

Yu Rong, Jingbo Wang, Ziwei Liu, Chen Change Loy

Abstract

Reconstructing the 3D geometry of two interacting hands from a single RGB image is a significant challenge due to severe occlusions and complex hand-to-hand interactions. Existing methods often produce physically impossible "collisions" where reconstructed meshes overlap. We present IHMR (Interacting Hand Mesh Recovery), a framework that decouples pose estimation from collision avoidance through a two-stage approach. By first predicting a coarse pose and then applying a factorized refinement strategy across translation, orientation, shape, and finger poses, our method significantly reduces mesh penetrations while improving overall pose accuracy. Our approach achieves state-of-the-art performance on the InterHand2.6M dataset, with an MLP-based variant capable of real-time execution at nearly 30 FPS.

Introduction

While 3D reconstruction of isolated hands has matured, reconstructing interacting hands remains difficult. When standard single-hand models are applied to two-hand scenarios, they often treat the hands as independent entities. This lack of mutual awareness leads to significant mesh collisions—unnatural overlaps that violate physical constraints.

Figure 1: Comparison of interacting hand reconstruction. (a) Input image. (b) Baseline methods often result in mesh collisions (red circles). (c, d) The proposed refinement strategy progressively resolves these overlaps to produce physically plausible results.

Simply incorporating a collision loss during the training of a neural network often fails; the model typically learns a "lazy" solution by pulling the hands apart to minimize the loss, which inadvertently destroys the accuracy of the actual 3D pose. This "collision-pose trade-off" necessitates a method that can resolve physical inconsistencies without sacrificing visual alignment.

Methodology: The IHMR Framework

The proposed IHMR framework operates in two distinct stages to resolve the conflict between pose accuracy and collision avoidance.

Stage I: Coarse Prediction

In the first stage, a Convolutional Neural Network (CNN) takes a monocular RGB image and regresses the parameters for the MANO hand model. This stage is designed to focus exclusively on achieving the most accurate 3D pose possible based on visual evidence, intentionally ignoring potential collisions to avoid the "lazy solution" trap.

Stage II: Factorized Refinement

The core innovation lies in the factorized refinement stage. Rather than attempting to fix all errors simultaneously, the system decomposes the causes of collisions into four distinct factors: relative translation (wrist positioning), global orientation, hand shape (thickness), and finger poses (joint angles).

Figure 2: The IHMR pipeline. Stage I performs coarse pose and joint prediction. Stage II applies a sequence of factorized refinements to the MANO parameters to resolve collisions and improve fit.

These factors are refined sequentially. A verification module evaluates each candidate refinement to ensure it reduces collisions while maintaining joint accuracy. To prevent the mesh from drifting away from the image evidence during refinement, the system uses predicted 3D joint heatmaps from a secondary "Joint Head" as a reliable pseudo-ground-truth benchmark.

Two implementations of Stage II are provided:

  • IHMR-OPT: An optimization-based version that provides maximum accuracy but runs slowly (0.9 FPS).
  • IHMR-MLP: A fast version using Multilayer Perceptrons as proxies for the refinement steps, achieving real-time performance (29.6 FPS).

Experimental Results

The framework was primarily evaluated on the InterHand2.6M dataset, focusing on subsets with close hand interactions.

  • Collision Reduction: On the most challenging "Inter-Close" subset, IHMR-OPT reduced the Average Penetration Depth (AVE-P) by 71.4% compared to the baseline.
  • Pose Accuracy: The method improved interacting pose estimation accuracy (I-MPJPE) by 16.5%, demonstrating that resolving collisions also helps refine the overall 3D pose.
  • Generalization: The model showed robust performance when tested on external datasets like RGB2Hand and Tzionas, proving its ability to generalize beyond its training environment.

Conclusion and Future Work

IHMR successfully addresses the collision-pose trade-off in monocular interacting hand reconstruction. By factorizing the refinement process and utilizing a verification module, the framework produces physically plausible and visually accurate 3D meshes. While the current model excels at resolving overlaps, future work will focus on handling extreme occlusions where one hand is nearly hidden and incorporating temporal consistency for smoother video-based reconstructions.

Summary updated

BibTeX
@inproceedings{rong2021monocular,
  author = {Yu Rong and Jingbo Wang and Ziwei Liu and Chen Change Loy},
  title = {{Monocular 3D Reconstruction of Interacting Hands via Collision-Aware Factorized Refinements}},
  booktitle = {3DV},
  year = {2021},
  doi = {10.1109/3dv53792.2021.00053},
}
Figure