Abstract
Estimating 3D hand meshes from single RGB images is challenging, due to intrinsic 2D-3D mapping ambiguities and limited training data. We adopt a compact parametric 3D hand model that represents deformable and articulated hand meshes. To achieve the model fitting to RGB images, we investigate and contribute in three ways: 1) Neural rendering: our hand mesh estimator (HME) is implemented by a neural network and a differentiable renderer, supervised by 2D segmentation masks and 3D skeletons. 2) Iterative testing refinement: we iteratively refine the initial estimate using gradients. 3) Self-data augmentation: we augment view-points on top of estimated dense hand poses. Experiments show our framework offers beyond state-of-the-art accuracy in 3D pose estimation and recovers dense 3D hand shapes.
Introduction
Estimating 3D hand poses from a single 2D image is an "ill-posed" problem—meaning many different 3D configurations could result in the same 2D projection. While most existing research focuses on "sparse" poses (identifying 21 key skeletal joints), many real-world applications like robotic grasping or AR/VR require "dense" information—the actual surface and volume of the hand. The primary challenge is the lack of high-quality training data; while 2D labels are accessible, perfect 3D mesh ground truth for real human hands in natural settings is extremely difficult to obtain.
Methodology
The proposed Dense Hand Pose Estimator (DHPE) framework bridges the gap between sparse 2D data and dense 3D reality through three core components:
- Neural Rendering & Parametric Modeling: The system utilizes the MANO model, a compact representation of hand shapes. Instead of requiring direct 3D mesh labels, the network is supervised via a differentiable renderer. This allows the model to project its 3D estimate back into a 2D mask and compare it against 2D evidence, such as segmentation masks and skeletons.
- Iterative Testing Refinement: Unlike standard feed-forward networks, DHPE uses gradients from its internal self-supervision to iteratively refine the mesh at test time, "snapping" the 3D model to better fit the observed 2D features.
- Self-Data Augmentation: To overcome data scarcity, the authors use fitted models to generate new training samples by re-rendering realistic hand fits from varied viewpoints and with different shapes.
Experiments and Results
The framework was evaluated on the Rendered Hand Pose Dataset (RHD), Stereo Hand Pose Dataset (SHD), and Dexter+Object (DO). A key insight is that jointly estimating shape and pose improves skeletal joint accuracy, as geometric constraints prevent anatomically impossible configurations.
Quantitatively, the method achieved state-of-the-art results. On the RHD dataset, the model showed a significant accuracy gain of approximately 4mm over prior 2.5D depth-map guided methods. Furthermore, the DHPE produced cleaner, more anatomically correct hand masks than dedicated segmentation networks, which often struggle with background clutter.
Conclusion
By combining parametric modeling with differentiable rendering and iterative refinement, this work pushes the boundaries of dense 3D hand reconstruction from RGB images. While self-occlusion and complex object interactions remain challenging, the framework's ability to recover accurate 3D shapes from sparse 2D supervision marks a significant step forward for applications in robotics and immersive technologies.
Summary updated