HandFlow: Quantifying View-Dependent 3D Ambiguity in Two-Hand Reconstruction with Normalizing Flow

Jiayi Wang, Diogo C. Luvizon, Franziska Mueller, Florian Bernard, Adam Kortylewski, Dan Casas, Christian Theobalt

Abstract

Monocular 3D reconstruction of interacting hands is a challenging task due to inherent projective ambiguity and frequent occlusions. Traditional deterministic methods often fail by predicting a single, potentially physically impossible "average" pose. We present HandFlow, a probabilistic framework that leverages Conditional Normalizing Flows to model the full distribution of plausible 3D hand poses. By introducing the MultiHands dataset—which provides multiple valid annotations per image—we demonstrate that HandFlow effectively quantifies uncertainty and outperforms existing probabilistic baselines.

Introduction

The problem of reconstructing two interacting hands from a single RGB image is "ill-posed." Because a 2D image loses depth information, multiple 3D configurations can result in the same 2D projection. Furthermore, hands frequently overlap, hiding critical joints from the camera's view. Most existing AI models are deterministic, attempting to map an image to a single "best" 3D pose. When faced with ambiguity, these models often produce an average of possible poses or fail entirely under heavy occlusion. HandFlow shifts this paradigm by embracing ambiguity as a feature, modeling a distribution of plausible poses rather than a single point estimate.

Methodology: HandFlowNet

The core of our approach is HandFlowNet, which outputs a distribution of plausible 3D hand poses conditioned on 2D image features.

Feature Extraction and Hand Modeling

We utilize a ResNet-50 backbone to extract 2D cues from the input image into a feature vector. For hand representation, we employ the MANO parametric hand model, which provides a robust basis for hand shape and pose.

Conditional Normalizing Flow

The heart of HandFlowNet is a conditional GLOW architecture. Normalizing flows learn an invertible mapping between a simple distribution (such as a Gaussian) and the complex, multi-modal distribution of 3D hand poses. This allows the model to sample numerous valid poses for a single input image, capturing the range of uncertainty inherent in the view.

Visualization of the joint distribution learned by HandFlow. The overlapping skeletons represent the range of plausible 3D poses captured by the normalizing flow, illustrating the model's ability to quantify spatial uncertainty.

Determinant Magnitude Regularization

A significant challenge in training normalizing flows is "mode collapse," where the model's output lacks diversity. To combat this, we introduced a novel loss term, Determinant Magnitude Regularization ($L_{DetMag}$). This term encourages the distribution to remain spread out and diverse while ensuring all samples remain consistent with the 2D image evidence.

The MultiHands Dataset

To evaluate a probabilistic model, one requires more than a single ground-truth label. We recognized that traditional datasets like InterHand2.6M are insufficient for this purpose. Consequently, we developed MultiHands, an extension of InterHand2.6M that provides 100 additional plausible annotations per image. These annotations were generated using a perturbation and plausibility-checking pipeline, ensuring that every pose is anatomically correct and free of hand-to-hand interpenetration.

Results and Discussion

We propose using Maximum Mean Discrepancy (MMD) as a primary evaluation metric, as it compares the entire predicted distribution against the distribution of multiple ground-truth annotations.

Quantitative and Qualitative Performance

HandFlow achieved an MMD of 0.50 (Global) and 0.42 (Relative), significantly outperforming other probabilistic baselines such as Variational Autoencoders (0.61) and MC-Dropout (0.91). Qualitative results demonstrate that sampled poses project perfectly onto the 2D image but vary significantly in 3D depth and rotation—precisely the behavior expected from monocular ambiguity.

Qualitative results of HandFlowNet. From a single camera view (left), the model generates multiple plausible 3D reconstructions (middle and right) that both align with the observed 2D evidence despite significant occlusion.

Application: View Selection

Because HandFlow quantifies its own uncertainty, it can be used to optimize multi-camera setups. By selecting the camera view with the lowest "sample variance," the system can automatically prioritize the angle that offers the clearest, least ambiguous view of the interacting hands.

Conclusion

HandFlow demonstrates that modeling the inherent ambiguity of 3D hand reconstruction leads to more robust and informative results. By moving away from deterministic point estimates and toward probabilistic distributions, we provide a framework that better captures the reality of complex hand interactions. Future work will focus on incorporating explicit physical constraints to further refine the plausibility of sampled poses.

Summary updated

Abstract

Reconstructing two-hand interactions from a single image is a challenging problem due to ambiguities that stem from projective geometry and heavy occlusions. Existing methods are designed to estimate only a single pose, despite the fact that there exist other valid reconstructions that fit the image evidence equally well. In this paper we propose to address this issue by explicitly modeling the distribution of plausible reconstructions in a conditional normalizing flow framework. This allows us to directly supervise the posterior distribution through a novel determinant magnitude regularization, which is key to varied 3D hand pose samples that project well into the input image. We also demonstrate that metrics commonly used to assess reconstruction quality are insufficient to evaluate pose predictions under such severe ambiguity. To address this, we release the first dataset with multiple plausible annotations per image called MultiHands. The additional annotations enable us to evaluate the estimated distribution using the maximum mean discrepancy metric. Through this, we demonstrate the quality of our probabilistic reconstruction and show that explicit ambiguity modeling is better-suited for this challenging problem.
BibTeX
@inproceedings{wang2022handflow,
  author = {Jiayi Wang and Diogo C. Luvizon and Franziska Mueller and Florian Bernard and Adam Kortylewski and Dan Casas and Christian Theobalt},
  title = {{HandFlow: Quantifying View-Dependent 3D Ambiguity in Two-Hand Reconstruction with Normalizing Flow}},
  booktitle = {VMV},
  year = {2022},
  url = {https://openreview.net/forum?id=bBrqBD_91K},
}
Figure