Abstract
Accurate fingertip tracking is a cornerstone of effective Human-Computer Interaction (HCI). However, traditional methods often struggle with self-occlusion, sensor noise, and the accumulation of errors from hierarchical skeletal models. This paper presents a novel two-stream Convolutional Neural Network (CNN) that integrates raw depth data with edge information extracted via Random Forest. By employing a "slow fusion" architecture, the model achieves state-of-the-art accuracy on major benchmarks, including a 9.9mm mean 3D error on the HandNet dataset, while maintaining real-time performance.
Introduction
In the world of HCI, tracking hand movements is essential for gestures like swiping and tapping. Fingertips are notoriously difficult to track accurately because they reside at the end of the hand's skeletal chain; errors in palm or wrist estimation accumulate, often leading to fingertip position errors exceeding 1cm. Furthermore, fingertips frequently suffer from self-occlusion and sensor noise, where depth cameras struggle to capture the thin, sharp edges of fingers.
Methodology
The proposed system treats depth and edges as distinct but complementary signals to improve localization robustness.
Edge Extraction and Two-Stream Architecture
The pipeline begins with a Random Forest (RF) classifier that processes raw depth images to extract clean edge maps, highlighting hand contours that are often blurred in raw data. These two inputs—depth and edges—are then fed into a two-stream CNN.
The researchers explored several fusion strategies:
- Early Fusion: Merging inputs into a two-channel image at the very beginning.
- Late Fusion: Keeping streams entirely separate until the final decision layers.
- Slow Fusion: The optimal strategy, where the first five convolutional layers share parameters to learn basic geometric features before splitting into specialized streams and finally merging at the sixth layer.
Joint Regression
Instead of predicting fingertips in isolation, the network performs joint regression, outputting an 18-element vector representing the 3D coordinates of five fingertips plus the palm. The palm acts as a stable spatial reference point (a "prior") that helps the network constrain where the fingertips should be.
Key Results
The model was evaluated on two major benchmarks: the primary HandNet dataset (214,000+ images) and the challenging NYU Hand Dataset.
- Accuracy: On HandNet, the proposed "CNN-fusion-slow" model achieved a 9.9mm mean 3D error, significantly outperforming the previous state-of-the-art error of 23.4mm. On the NYU dataset, it achieved a 19.3mm error, surpassing other prominent architectures like DeepPrior (26.4mm).
- Efficiency: Despite the two-stream complexity, the architecture reaches 25 FPS on a standard CPU, making it suitable for real-time interactive applications without requiring high-end GPUs for inference.
Discussion and Conclusion
The success of this approach highlights the value of using edges as a geometric prior. While CNNs can theoretically learn to find edges, explicitly providing an edge map gives the network a massive "head start," especially in noisy areas. The "slow fusion" timing is also critical, balancing specialized feature extraction with the integration of correlated depth and shape data. Future work may extend this fusion technique to full-body human pose estimation or more complex object recognition tasks where depth boundaries are critical.
Summary updated