Abstract
Deterministic models for 3D hand pose reconstruction, whether single-staged or cascaded, struggle with pose ambiguities caused by self-occlusions and complex hand articulations. Existing cascaded approaches refine predictions in a coarse-to-fine manner but remain deterministic and cannot capture pose uncertainties. Recent probabilistic methods model pose distributions yet are restricted to single-stage estimation, which often fails to produce accurate 3D reconstructions without refinement. To address these limitations, we propose a coarse-to-fine cascaded diffusion framework that combines probabilistic modeling with cascaded refinement. The first stage is a joint diffusion model that samples diverse 3D joint hypotheses, and the second stage is a Mesh Latent Diffusion Model (Mesh LDM) that reconstructs a 3D hand mesh conditioned on a joint sample. By training Mesh LDM with diverse joint hypotheses in a learned latent space, our framework learns distribution-aware joint-mesh relationships and robust hand priors. Furthermore, the cascaded design mitigates the difficulty of directly mapping 2D images to dense 3D poses, enhancing accuracy through sequential refinement. Experiments on FreiHAND and HO3Dv2 demonstrate that our method achieves state-of-the-art performance while effectively modeling pose distributions.
Introduction
Estimating the 3D shape and pose of a hand from a single 2D image is a fundamental challenge in computer vision. Hands are highly articulated and frequently subject to self-occlusion, where fingers are hidden behind the palm or other fingers. Traditional deterministic models attempt to predict a single "correct" pose, which often leads to physically impossible or "averaged" results when faced with visual ambiguity. While probabilistic models have been introduced to suggest multiple plausible poses, single-stage approaches often struggle to map raw pixels directly to dense 3D meshes accurately.
Proposed Framework
To overcome these hurdles, we introduce a coarse-to-fine cascaded architecture utilizing denoising diffusion models. The framework decomposes the reconstruction task into two distinct probabilistic stages:
- Stage 1: Joint Diffusion Model (Coarse): The model first "uplifts" 2D keypoints into a set of 3D joint hypotheses. As a diffusion-based model, it generates a distribution of plausible skeletons that align with the 2D evidence rather than a single regressed point.
- Stage 2: Mesh Latent Diffusion Model (Fine): This stage consumes a 3D joint sample from the first stage along with original image features to reconstruct a high-fidelity 3D mesh.
Latent Space Innovation
A critical component of this framework is the space in which diffusion occurs. Standard methods often diffuse in raw 3D coordinate space, which can lose surface geometry, or rely on rigid parametric models like MANO, which may restrict realistic movement. Our approach performs diffusion in a learned latent space via a Mesh AutoEncoder. This allows the model to maintain global hand structure while capturing expressive, non-linear surface deformations and fine-grained details.
By training the Mesh LDM on the diverse, noisy hypotheses generated by the Joint Diffusion stage (Distribution-Aware Training), the model becomes exceptionally robust. It learns to effectively "fix" or interpret various plausible skeleton inputs to produce a coherent mesh.
Experimental Evaluation
The framework was evaluated on two benchmark datasets:
- FreiHAND: A large-scale dataset of single hands (133K images). Our model achieved a Procrustes-aligned Mean Per Joint Position Error (P-MPJPE) of 4.4mm (Best-of-N), setting a new state-of-the-art.
- HO3Dv2: A challenging dataset featuring hand-object interactions and significant occlusion (66K images). The model achieved a P-MPJPE of 7.4mm, demonstrating strong generalization in complex scenarios.
Qualitatively, the model excels in ambiguous poses. For instance, in side-view "rock" gestures where fingers are occluded, our framework produces natural finger bending, whereas competing models often yield stiff or unnatural results.
Conclusion
This cascaded diffusion framework successfully bridges the gap between probabilistic modeling and high-fidelity 3D reconstruction. While the iterative nature of diffusion models presents a challenge for real-time inference speed, the accuracy and robustness gained—especially in occluded scenarios—mark a significant advancement. Future work will focus on extending this framework to multi-hand interactions and optimizing computational efficiency for broader deployment.
Summary updated