Interactive Hand Pose Estimation: Boosting accuracy in localizing extended finger joints

Cairong Zhang, Guijin Wang, Hengkai Guo, Xinghao Chen, Fei Qiao, Huazhong Yang

Introduction

In the evolving landscape of Human-Machine Interaction (HMI), precision in gesture recognition is paramount. While depth-based hand pose estimation has seen significant progress, current models often struggle with the most critical components of interaction: the fingertips. Fingertips possess high degrees of freedom and frequently suffer from "part confusion," where one finger is mistaken for another. This "accuracy gap" between the stable palm and the highly mobile fingertips often leads to failed interactions in VR, AR, and gesture-controlled systems. Researchers from Tsinghua University have addressed this by introducing Interactive Hand Pose Estimation (IHPE), a refinement scheme specifically designed to localize extended finger joints with high precision.

The IHPE Pipeline

The proposed IHPE system operates as a two-stage pipeline that prioritizes computational efficiency, allowing it to run in real-time on standard CPU hardware.

Stage 1: Baseline Estimation

The process begins with a Cascaded Hierarchical Regression model. This initial stage provides a "rough" estimate of the hand's pose by first locating the palm and then estimating finger positions relative to that center. While effective for the general hand shape, this baseline often lacks the precision required for fine-grained fingertip tasks.

Stage 2: Refinement through Neighbor Pixel Voting

The core innovation lies in the refinement stage, which focuses on "stretching-out" fingers. This stage involves three distinct steps:

  1. Detection: Using hand masks and distance transforms, the system identifies which fingers are currently extended.
  2. Identification: These detected extensions are matched to the rough baseline prediction to correctly label each finger (e.g., distinguishing the index from the middle finger).
  3. Neighbor Pixel Voting: For every joint in an extended finger, the system utilizes a Random Forest based on Pixel Difference Features. Nearby "foreground" pixels vote on the joint's actual location, and the final position is determined by averaging these votes.

By leveraging local context—where pixels near the finger's edge provide highly informative votes for the joint center—the model "drags" joints with large baseline errors toward their true anatomical locations.

Experimental Results

The IHPE algorithm was evaluated on two major public benchmarks: the MSRA15 Hand Pose Dataset and the ICVL Hand Pose Dataset.

The refinement stage yielded substantial improvements in localization accuracy. On the MSRA15 dataset, the mean error for all fingertips was reduced by 5.14mm (20.3%). More impressively, for fingers specifically identified as "stretching out," the fingertip error dropped by 10.51mm, representing a 41.4% relative improvement over the baseline.

Beyond accuracy, the system maintains a high throughput of 18 frames per second (fps) on a single CPU thread. This performance is achieved by avoiding heavy deep neural networks in favor of efficient Random Forests and simple depth-difference calculations, making it suitable for mobile AR/VR hardware.

Conclusion and Future Directions

The IHPE framework successfully bridges the accuracy gap for fingertip localization in interactive scenarios. By focusing refinement efforts on extended fingers and using local pixel voting, the researchers have created a more stable and reliable hand model for gesture-based interfaces.

While highly effective, the authors noted minor limitations, such as occasional confusion when the thumb joint overlaps with the palm. Future work will explore weighted voting mechanisms to give more confident pixels a larger influence and further improve the initial detection of extended fingers to ensure robust performance across all hand poses.

Summary updated

Abstract

Accurate 3D hand pose estimation plays an important role in Human Machine Interaction (HMI). In the reality of HMI, joints in fingers stretching out, especially corresponding fingertips, are much more important than other joints. We propose a novel method to refine stretching-out finger joint locations after obtaining rough hand pose estimation. It first detects which fingers are stretching out, then neighbor pixels of certain joint vote for its new location based on random forests. The algorithm is tested on two public datasets : MSRA15 and ICVL. After the refinement stage of stretching-out fingers, errors of predicted HMI finger joint locations are significantly reduced. Mean error of all fingertips reduces around 5mm (relatively more than 20%). Stretching-out fingertip locations are even more precise, which in MSRA15 reduces 10.51mm (relatively 41.4%).
BibTeX
@inproceedings{zhang2018interactive,
  author = {Cairong Zhang and Guijin Wang and Hengkai Guo and Xinghao Chen and Fei Qiao and Huazhong Yang},
  title = {{Interactive Hand Pose Estimation: Boosting accuracy in localizing extended finger joints}},
  booktitle = {Visual Information Processing and Communication},
  year = {2018},
  doi = {10.2352/issn.2470-1173.2018.2.vipc-251},
}
Figure