Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Rendering

Seungryul Baek, Kwang In Kim, Tae-Kyun Kim

Abstract

Estimating 3D hand meshes from single RGB images is challenging due to intrinsic 2D-3D mapping ambiguities and limited training data. We adopt a compact parametric 3D hand model that represents deformable and articulated hand meshes. To achieve model fitting to RGB images, we investigate and contribute in three ways: 1) Neural rendering: our hand mesh estimator (HME) is implemented by a neural network and a differentiable renderer, supervised by 2D segmentation masks and 3D skeletons. 2) Iterative testing refinement: we iteratively refine the initial estimate using gradients of a differentiable fitting function. 3) Self-data augmentation: we augment viewpoints on top of estimated dense hand poses to create new training data. Experiments using three RGB-based benchmarks show that our framework offers beyond state-of-the-art accuracy in 3D pose estimation and recovers dense 3D hand shapes.

Introduction

Estimating the 3D pose of a hand from a standard 2D image is a classic computer vision challenge. Most existing methods focus on "sparse" poses—essentially a stick-figure skeleton of 21 joints. While useful, skeletons lack the surface information needed for realistic AR/VR interactions or robotic grasping. The challenge is twofold: 2D images lack inherent depth, making many 3D configurations look identical from one angle, and it is difficult to collect ground truth 3D mesh data for real-world RGB images.

Methodology

The proposed Dense Hand Pose Estimator (DHPE) addresses these issues through three main technical pillars:

1. Neural Rendering (The HME)

Instead of predicting a mesh directly, the Hand Mesh Estimator (HME) predicts the parameters of the MANO model, a compact mathematical representation of hand shape and pose. As illustrated in the architecture diagram below, a differentiable renderer projects this 3D model back into a 2D segmentation mask, allowing the network to learn from 2D labels which are much easier to obtain than 3D meshes.

Architecture of the Dense Hand Pose Estimator (DHPE) showing the 2D evidence and 3D mesh estimation stages.

2. Iterative Testing Refinement

Inspired by classical alignment algorithms like Iterative Closest Point (ICP), the framework does not rely solely on the network's initial prediction. At test time, the system uses the gradients of the differentiable fitting function to iteratively "nudge" the predicted parameters until the projected 3D mesh aligns perfectly with the 2D evidence in the image. This hierarchical optimization first focuses on articulation before refining the hand's shape.

3. Self-supervised Data Augmentation

To overcome data scarcity, the model "recycles" successful fits. By taking a fitted 3D hand and tweaking its viewpoint and shape, the authors render new synthetic training examples. This creates a feedback loop that improves generalization to diverse hand shapes and camera angles.

Experimental Results

The model was evaluated on three major benchmarks: the Rendered Hand Pose Dataset (RHD), the Stereo Hand Pose Dataset (SHD), and the Dexter+Object (DO) dataset.

Qualitative results across RHD, DO, and SHD datasets, showing input images, skeleton overlays, and recovered 3D meshes.

Key Findings

  • State-of-the-Art Accuracy: The method significantly outperformed previous approaches, such as Zimmermann & Brox. On the RHD dataset, it achieved a higher Area Under the Curve (AUC) for 3D joint accuracy.
  • Robustness: The system proved highly effective at segmenting hands from cluttered backgrounds, outperforming dedicated segmentation algorithms that often struggle with skin-colored objects.
  • Refinement Gains: The iterative refinement step provided a noticeable boost in accuracy, particularly on the real-world Dexter+Object dataset, proving its value for generalization.

Discussion and Limitations

While the framework pushes the envelope for accuracy, the iterative refinement process adds computational overhead. The initial regression takes ~100ms, but the refinement iterations add another 250ms, bringing the total pipeline to approximately 350ms per frame. Furthermore, while the MANO prior helps with self-occlusion, heavy occlusion by external objects remains a challenging area for future research.

Summary updated

Abstract

Estimating 3D hand meshes from single RGB images is challenging, due to intrinsic 2D-3D mapping ambiguities and limited training data. We adopt a compact parametric 3D hand model that represents deformable and articulated hand meshes. To achieve the model fitting to RGB images, we investigate and contribute in three ways: 1) Neural rendering: inspired by recent work on human body, our hand mesh estimator (HME) is implemented by a neural network and a differentiable renderer, supervised by 2D segmentation masks and 3D skeletons. HME demonstrates good performance for estimating diverse hand shapes and improves pose estimation accuracies. 2) Iterative testing refinement: Our fitting function is differentiable. We iteratively refine the initial estimate using the gradients, in the spirit of iterative model fitting methods like ICP. The idea is supported by the latest research on human body. 3) Self-data augmentation: collecting sized RGB-mesh (or segmentation mask)-skeleton triplets for training is a big hurdle. Once the model is successfully fitted to input RGB images, its meshes i.e. shapes and articulations, are realistic, and we augment view-points on top of estimated dense hand poses. Experiments using three RGB-based benchmarks show that our framework offers beyond state-of-the-art accuracy in 3D pose estimation, as well as recovers dense 3D hand shapes. Each technical component above meaningfully improves the accuracy in the ablation study.
BibTeX
@inproceedings{baek2019pushing,
  author = {Seungryul Baek and Kwang In Kim and Tae-Kyun Kim},
  title = {{Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Rendering}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  url = {https://openreview.net/forum?id=sjHuJJZKU6v},
}

Loading related content...

Figure