Capturing Hand Articulations using Recurrent Neural Network for 3D Hand Pose Estimation

Cheol-Hwan Yoo, Seung-Wook Kim, Seo-Won Ji, Yong-Goo Shin, Sung-Jea Ko

Abstract

3D hand pose estimation from a single depth image plays an important role in computer vision and human-computer interaction. Although recent hand pose estimation methods using convolution neural network (CNN) have shown notable improvements in accuracy, most of them have a limitation that they rely on a complex network structure without fully exploiting the articulated structure of the hand. A hand, which is an articulated object, is composed of six local parts: the palm and five independent fingers. Each finger consists of sequential-joints that provide constrained motion, referred to as a kinematic chain. In this paper, we propose a hierarchically-structured convolutional recurrent neural network (HCRNN) with six branches that estimate the 3D position of the palm and five fingers independently. The palm position is predicted via fully-connected layers. Each sequential-joint, i.e. finger position, is obtained using a recurrent neural network (RNN) to capture the spatial dependencies between adjacent joints. Then the output features of the palm and finger branches are concatenated to estimate the global hand position. HCRNN directly takes the depth map as an input without a time-consuming data conversion, such as 3D voxels and point clouds. Experimental results on public datasets demonstrate that the proposed HCRNN not only outperforms most 2D CNN-based methods using the depth image as their inputs but also achieves competitive results with state-of-the-art 3D CNN-based methods with a highly efficient running speed of 285 fps on a single GPU.

Introduction

Predicting the exact 3D coordinates of hand joints from a single depth image is a fundamental challenge in computer vision. The complexity arises from the hand's high degrees of freedom, frequent self-occlusions, and varied orientations. Existing approaches generally fall into two categories: 3D-based methods and 2D-based methods.

3D-based methods typically convert 2D depth maps into volumetric representations like voxels or point clouds. While these methods achieve high accuracy by preserving spatial information, they are computationally expensive and slow due to the required data conversion and 3D convolutions. Conversely, 2D-based methods are faster but often treat hand joints as independent points, neglecting the physical constraints of the hand's articulated structure.

This research introduces the Hierarchically-structured Convolutional Recurrent Neural Network (HCRNN), a model designed to bridge the gap between efficiency and structural awareness.

Methodology: The HCRNN Architecture

The core innovation of HCRNN lies in its hierarchical structure, which treats human fingers as sequential "kinematic chains." The architecture consists of two primary components: an encoder and a regression subnet.

Architecture of the proposed HCRNN model, showing the encoder and the six-branch regression subnet.

The Encoder

The encoder is a shallow ResNet-based network that extracts low-level features directly from the 2D depth map. A shallow design is preferred because depth maps typically contain simpler textures compared to RGB images, allowing for faster processing without significant loss of relevant information.

The Regression SubNet

The regression subnet implements the hierarchical approach by splitting into six distinct branches: one for the palm and five for the fingers.

  • Palm Branch: Uses fully-connected layers to predict the palm position, providing a stable global reference.
  • Finger Branches: Each finger is modeled as a sequence of joints. The researchers utilized Gated Recurrent Units (GRUs) to process joint features sequentially. This mimics the physical reality where a fingertip's position is spatially dependent on the preceding joints (knuckles).
  • Feature Fusion: The features from all six branches are concatenated to produce the final global hand pose estimation.

By processing raw depth maps directly, HCRNN avoids the latency associated with 3D voxelization while still capturing essential 3D spatial relationships through its recurrent architecture.

Experimental Setup

The HCRNN model was rigorously evaluated on three industry-standard benchmarks:

  1. NYU Hand Pose Dataset: Contains 72,000 training and 8,000 testing images, characterized by high noise and complex poses.
  2. ICVL Hand Posture Dataset: Includes 22,000 training frames with high-quality depth maps and minimal noise.
  3. MSRA Hand Gesture Dataset: Comprises 76,000 frames from nine subjects, testing the model's generalization across different hand sizes and shapes.

Results and Discussion

The experimental results highlight a significant breakthrough in the speed-accuracy trade-off for hand pose estimation.

Comparison of mean distance error versus inference time for various state-of-the-art methods.

Performance Metrics

  • Inference Speed: HCRNN achieved a remarkable speed of 285 FPS on a single NVIDIA Titan X GPU. This is approximately 80 times faster than V2V-PoseNet, a leading 3D-based model.
  • Accuracy: Despite its speed, HCRNN remains highly competitive. It achieved mean 3D distance errors of 6.54mm on ICVL, 9.37mm on NYU, and 7.70mm on MSRA. On the NYU dataset, it outperformed nearly all other 2D-based methods.
  • Ablation Studies: The researchers found that the hierarchical six-branch structure reduced error by 5% compared to a two-branch model. Furthermore, using RNNs for finger joints instead of standard fully-connected layers provided an additional 7% reduction in error.

Key Insights

The success of HCRNN stems from two main ideas. First, treating fingers as sequences allows the model to leverage physical constraints similar to how NLP models leverage linguistic context. Second, the "divide and conquer" strategy of using six branches limits the search space for the network, allowing for more specialized and accurate feature extraction.

Conclusion and Future Work

HCRNN demonstrates that sophisticated 3D spatial relationships can be extracted from 2D data without the need for computationally heavy 3D representations. While it slightly lags behind the absolute accuracy of the most complex 3D volumetric models, its superior speed makes it ideal for real-time applications in AR/VR and human-computer interaction.

Future research will focus on refining the ensemble strategy for combining branch features and exploring more complex backbones to further narrow the accuracy gap without compromising the real-time performance that defines HCRNN.

Summary updated

Abstract

3D hand pose estimation from a single depth image plays an important role in computer vision and human-computer interaction. Although recent hand pose estimation methods using convolution neural network (CNN) have shown notable improvements in accuracy, most of them have a limitation that they rely on a complex network structure without fully exploiting the articulated structure of the hand. A hand, which is an articulated object, is composed of six local parts: the palm and five independent fingers. Each finger consists of sequential-joints that provide constrained motion, referred to as a kinematic chain. In this paper, we propose a hierarchically-structured convolutional recurrent neural network (HCRNN) with six branches that estimate the 3D position of the palm and five fingers independently. The palm position is predicted via fully-connected layers. Each sequential-joint, i.e. finger position, is obtained using a recurrent neural network (RNN) to capture the spatial dependencies between adjacent joints. Then the output features of the palm and finger branches are concatenated to estimate the global hand position. HCRNN directly takes the depth map as an input without a time-consuming data conversion, such as 3D voxels and point clouds. Experimental results on public datasets demonstrate that the proposed HCRNN not only outperforms most 2D CNN-based methods using the depth image as their inputs but also achieves competitive results with state-of-the-art 3D CNN-based methods with a highly efficient running speed of 285 fps on a single GPU.
BibTeX
@inproceedings{yoo2020capturing,
  author = {Cheol-Hwan Yoo and Seung-Wook Kim and Seo-Won Ji and Yong-Goo Shin and Sung-Jea Ko},
  title = {{Capturing Hand Articulations using Recurrent Neural Network for 3D Hand Pose Estimation}},
  booktitle = {IEEE Access},
  year = {2020},
  url = {https://openreview.net/forum?id=uaIE3WmylO},
}
Figure