Interactive Hand Pose Estimation: Boosting accuracy in localizing extended finger joints

Cairong Zhang, Guijin Wang, Hengkai Guo, Xinghao Chen, Fei Qiao, Huazhong Yang

TL;DR

This paper introduces a targeted refinement stage for 3D hand pose estimation that specifically improves the localization of extended fingers—the most critical joints for human-machine interaction (HMI). By combining a baseline hierarchical regression with a novel neighbor-pixel voting scheme based on random forests, the authors achieve up to a 41.4% reduction in fingertip localization error on standard benchmarks.

What Problem Does This Paper Address?

Accurate 3D hand tracking is the backbone of immersive VR and AR experiences. However, the most important parts of the hand for interaction—the fingertips—are often the hardest to track. Because fingertips have high degrees of freedom and are prone to self-occlusion or "part confusion" (where the system mistakes one finger for another), they typically exhibit the highest estimation errors in standard models.

For HMI tasks like clicking or zooming, a few millimeters of error can mean the difference between a seamless experience and a frustrating one. This paper tackles the "accuracy gap" between palm joints and fingertips by focusing refinement efforts specifically on fingers that are stretched out and ready for interaction.

Methods

The authors propose a two-step pipeline:

  1. Baseline Estimation: They first use a standard Cascaded Hierarchical Regression (based on prior work by Sun et al.) to generate a "rough" 3D pose. This model breaks the task into palm stages (wrist, finger roots) and finger stages to manage complexity.
  2. Stretching-Out Finger Refinement: This is the core contribution. It involves:
  3. Detection: Using hand mask images and distance transforms, the system identifies which fingers are extended. It finds the palm center and then looks for local maxima on the hand boundary to locate potential fingertips.
  4. Identification: It matches these detected extended fingers to the baseline's rough predictions to determine which finger is which (e.g., index vs. ring).
  5. Neighbor Pixels Voting: For each joint in an extended finger, foreground pixels in its immediate vicinity "vote" for its correct 3D position. These votes are generated by a Random Forest trained on simple, computationally cheap depth difference features. The final joint position is the average of all these local votes.
The process of detecting stretching-out fingers from hand masks and distance transforms.

Datasets

The algorithm was evaluated on two widely used public benchmarks:

  • MSRA15: Contains 76.5K depth images featuring 9 subjects performing 17 different gestures.
  • ICVL: Includes approximately 22K training images and 1.6K testing images across 10 subjects and 26 gestures.

Key Insights and Ideas

  • Prioritizing Interaction: The authors recognize that in HMI, not all joints are created equal. By focusing computational resources on refining extended fingers, they improve the system where it matters most for the end-user.
  • Local vs. Global Information: While global regression is good for overall hand structure, local pixel-level voting is far more effective at "pulling" a joint toward its true anatomical center within a finger's contour.
  • Efficiency Matters: By using Random Forests and simple depth difference features, the entire system—including the refinement stage—runs at 18 FPS on a standard CPU. This makes it practical for real-time applications without requiring expensive GPUs.

Key Results

The refinement stage consistently outperformed the baseline across both datasets:

  • Fingertip Accuracy: On the MSRA15 dataset, the mean error for all fingertips was reduced by about 5mm (a >20% improvement).
  • Extended Finger Precision: The most dramatic gains were seen in stretching-out fingertips, where the error on MSRA15 dropped from 25.39mm to 14.88mm—a 41.4% relative improvement.
  • Overall Error: On the ICVL dataset, the mean error for all joints improved from 15.07mm to 12.98mm.
Comparison of joint localization error between the baseline and the proposed refinement method.

Qualitatively, the refinement stage effectively corrects "drifting" joints, ensuring they stay within the visible boundaries of the fingers in the depth image.

Limitations and Future Work

While the improvements are significant, the authors acknowledge that even higher precision (ideally sub-10mm error for fingertips) is needed for truly delicate HMI tasks.

Future research directions include:

  • Improving the initial detection accuracy of stretching-out fingers to ensure the refinement stage is triggered correctly.
  • Moving beyond simple averaging in the voting stage by introducing a weighted voting mechanism, where pixels closer to the predicted joint or with higher confidence have more influence on the final result.

Summary updated

Abstract

Accurate 3D hand pose estimation plays an important role in Human Machine Interaction (HMI). In the reality of HMI, joints in fingers stretching out, especially corresponding fingertips, are much more important than other joints. We propose a novel method to refine stretching-out finger joint locations after obtaining rough hand pose estimation. It first detects which fingers are stretching out, then neighbor pixels of certain joint vote for its new location based on random forests. The algorithm is tested on two public datasets: MSRA15 and ICVL. After the refinement stage of stretching-out fingers, errors of predicted HMI finger joint locations are significantly reduced. Mean error of all fingertips reduces around 5mm (relatively more than 20%). Stretching-out fingertip locations are even more precise, which in MSRA15 reduces 10.51mm (relatively 41.4%).
BibTeX
@inproceedings{zhang2018interactive,
  author = {Cairong Zhang and Guijin Wang and Hengkai Guo and Xinghao Chen and Fei Qiao and Huazhong Yang},
  title = {{Interactive Hand Pose Estimation: Boosting accuracy in localizing extended finger joints}},
  booktitle = {Visual Information Processing and Communication},
  year = {2018},
  doi = {10.2352/issn.2470-1173.2018.2.vipc-251},
}
Figure