Abstract
Hand segmentation is a critical precursor to 3D hand pose estimation, yet traditional methods often rely on simplistic depth-thresholding assumptions that fail in complex, real-world environments. We present the Soft Proposal Segmentation Network (SPS-Net), a robust framework that combines deep learning-based spatial features with temporal tracking. By utilizing "soft" heat map proposals and Kalman filtering, SPS-Net achieves state-of-the-art accuracy (0.94 mIoU on the NYU dataset) and real-time performance (50 FPS), proving resilient in egocentric and multi-view scenarios where traditional methods falter.
Introduction
In the field of 3D hand pose estimation, isolating the hand from the background—hand segmentation—has frequently been treated as a trivial task. Researchers often employ a fixed depth threshold, assuming the hand is always the object closest to the camera. However, this "closest-object" assumption is fragile. In realistic applications like Augmented Reality (AR) or egocentric (first-person) robotics, a user's arm, torso, or nearby furniture may exist at a similar depth to the hand, leading to catastrophic segmentation failures. When segmentation fails, subsequent finger-joint identification becomes nearly impossible.
SPS-Net addresses these challenges by moving beyond single-frame depth heuristics. Instead, it leverages temporal consistency—the principle that a hand's position in a video sequence is highly correlated across consecutive frames—to maintain robust tracking and segmentation even in cluttered or ambiguous scenes.
The SPS-Net Architecture
The Soft Proposal Segmentation Network is composed of five integrated modules designed to pass probabilistic information through the pipeline, avoiding the rigid errors common in traditional object detection.
- Feature Extraction Module (FEM): A CNN backbone utilizing Residual Blocks to extract high-level spatial features from raw depth input.
- Soft Proposal Generator: Rather than predicting a "hard" binary bounding box, this module generates a soft heat map ($H^d_i$) representing the spatial likelihood of the hand's location.
- Temporal Tracking: To ensure stability, a Kalman filter tracks the hand's center across frames. This produces a second soft proposal ($H^t_i$) based on the hand's predicted trajectory.
- Merging Module (MM): This module fuses the detection and tracking proposals. If a single frame is noisy or the hand is partially occluded, the temporal tracking "reminds" the network of the hand's most probable location.
- Mask Generator: A U-Net-inspired decoder uses the merged proposal and extracted features to generate the final high-resolution binary mask.
By using soft proposals (heat maps) instead of hard bounding boxes, the network prevents the accidental clipping of fingers—a frequent failure mode in rigid detection frameworks like Mask R-CNN.
Experimental Evaluation
The researchers evaluated SPS-Net across three diverse benchmarks to test its versatility:
- NYU Hand Dataset: Features a wide variety of poses across front, left, and right views with complex backgrounds.
- CVAR Dataset: Focuses on egocentric views, which are particularly challenging for depth-thresholding due to the proximity of the user's body.
- Hand2017 Challenge Dataset: A large-scale dataset used for global 3D Hand Pose Tracking benchmarks.
Key Results and Discussion
SPS-Net demonstrates significant improvements over baseline segmentation models in both accuracy and speed:
- Superior Accuracy: On the NYU dataset, SPS-Net achieved a mean Intersection over Union (mIoU) of 0.94, outperforming standard U-Net (0.87) and Mask R-CNN (0.82).
- Benchmark Performance: When integrated with a 3D pose estimator, the system secured 1st place in the Hand2017 Challenge, recording an Average Euclidean Error (ADE) of only 10.48 mm.
- Real-Time Efficiency: The model maintains a processing speed of 50 FPS on an NVIDIA Titan X, making it viable for low-latency VR/AR applications.
- Ablation Insights: Testing confirmed that the fusion of detection and temporal tracking proposals is essential; the combined approach consistently outperformed models relying on only one of the two components.
Conclusion
SPS-Net proves that temporal information and soft spatial proposals are vital for reliable hand segmentation in depth videos. By moving away from rigid depth thresholds, the model provides a stable foundation for complex downstream tasks such as sign language interpretation and human-computer interaction. Future work may focus on improving tracking recovery during extremely high-speed movements or prolonged occlusions where the Kalman filter's initialization might be challenged.
Summary updated