Abstract
Monocular 3D reconstruction of interacting hands is a challenging task due to inherent projective ambiguity and frequent occlusions. Traditional deterministic methods often fail by predicting a single, potentially physically impossible "average" pose. We present HandFlow, a probabilistic framework that leverages Conditional Normalizing Flows to model the full distribution of plausible 3D hand poses. By introducing the MultiHands dataset—which provides multiple valid annotations per image—we demonstrate that HandFlow effectively quantifies uncertainty and outperforms existing probabilistic baselines.
Introduction
The problem of reconstructing two interacting hands from a single RGB image is "ill-posed." Because a 2D image loses depth information, multiple 3D configurations can result in the same 2D projection. Furthermore, hands frequently overlap, hiding critical joints from the camera's view. Most existing AI models are deterministic, attempting to map an image to a single "best" 3D pose. When faced with ambiguity, these models often produce an average of possible poses or fail entirely under heavy occlusion. HandFlow shifts this paradigm by embracing ambiguity as a feature, modeling a distribution of plausible poses rather than a single point estimate.
Methodology: HandFlowNet
The core of our approach is HandFlowNet, which outputs a distribution of plausible 3D hand poses conditioned on 2D image features.
Feature Extraction and Hand Modeling
We utilize a ResNet-50 backbone to extract 2D cues from the input image into a feature vector. For hand representation, we employ the MANO parametric hand model, which provides a robust basis for hand shape and pose.
Conditional Normalizing Flow
The heart of HandFlowNet is a conditional GLOW architecture. Normalizing flows learn an invertible mapping between a simple distribution (such as a Gaussian) and the complex, multi-modal distribution of 3D hand poses. This allows the model to sample numerous valid poses for a single input image, capturing the range of uncertainty inherent in the view.
Determinant Magnitude Regularization
A significant challenge in training normalizing flows is "mode collapse," where the model's output lacks diversity. To combat this, we introduced a novel loss term, Determinant Magnitude Regularization ($L_{DetMag}$). This term encourages the distribution to remain spread out and diverse while ensuring all samples remain consistent with the 2D image evidence.
The MultiHands Dataset
To evaluate a probabilistic model, one requires more than a single ground-truth label. We recognized that traditional datasets like InterHand2.6M are insufficient for this purpose. Consequently, we developed MultiHands, an extension of InterHand2.6M that provides 100 additional plausible annotations per image. These annotations were generated using a perturbation and plausibility-checking pipeline, ensuring that every pose is anatomically correct and free of hand-to-hand interpenetration.
Results and Discussion
We propose using Maximum Mean Discrepancy (MMD) as a primary evaluation metric, as it compares the entire predicted distribution against the distribution of multiple ground-truth annotations.
Quantitative and Qualitative Performance
HandFlow achieved an MMD of 0.50 (Global) and 0.42 (Relative), significantly outperforming other probabilistic baselines such as Variational Autoencoders (0.61) and MC-Dropout (0.91). Qualitative results demonstrate that sampled poses project perfectly onto the 2D image but vary significantly in 3D depth and rotation—precisely the behavior expected from monocular ambiguity.
Application: View Selection
Because HandFlow quantifies its own uncertainty, it can be used to optimize multi-camera setups. By selecting the camera view with the lowest "sample variance," the system can automatically prioritize the angle that offers the clearest, least ambiguous view of the interacting hands.
Conclusion
HandFlow demonstrates that modeling the inherent ambiguity of 3D hand reconstruction leads to more robust and informative results. By moving away from deterministic point estimates and toward probabilistic distributions, we provide a framework that better captures the reality of complex hand interactions. Future work will focus on incorporating explicit physical constraints to further refine the plausibility of sampled poses.
Summary updated