Introduction
Static gesture recognition—identifying a hand sign from a single snapshot—is a cornerstone of human-computer interaction. However, it faces a significant challenge: occlusion. When a user’s hand is not perfectly aligned with a single camera, fingers can be hidden behind the palm (self-occlusion), or external objects can block the view. To address this, researchers at UC Berkeley developed DuoDepth, a system that utilizes two synchronized depth cameras to eliminate the self-occlusion problem. By processing 3D point clouds from multiple angles, DuoDepth achieves a 53.4% reduction in misclassification compared to traditional single-camera setups.
Methodology
The DuoDepth system employs a multi-view approach using two Intel RealSense D415 depth cameras mounted perpendicular to each other. Raw depth data is transformed into lightweight, sparse 3D point clouds. To isolate the hand from the user's face and body, the system uses a peak detection algorithm on the Z-axis (depth) histogram. As shown in Figure 1, if multiple peaks are detected, the system isolates the arm and discards background data like the head or torso.
The researchers evaluated three neural network architectures based on PointNet:
- FUSED: Stitches two point clouds into one model using Iterative Closest Point (ICP) registration before classification.
- DUAL-FEAT: Processes each view through its own Spatial Transformer Network (STN3D) and fuses extracted features.
- SOLO-FEAT: Extracts features independently and combines them only at the final classifier stage.
A key finding in the system's design was the relationship between point density and performance. The team determined that only 320 points per gesture were necessary to maintain high accuracy (Figure 2). This sparsity allows the entire pipeline—including cropping, down-sampling, and classification—to run in just 7.1ms.
Dataset
The study utilized a custom dataset of 20,100 point clouds, covering 10 distinct gestures (Zero, One, Two, Three, Frame, Four, Five, Ell, Thumb, and Bird). Captures were varied by hand orientation relative to the cameras to ensure robustness. The authors have made the code and dataset available on GitHub.
Key Results and Insights
The DUAL-FEAT model emerged as the top performer, achieving a mean accuracy of 94.5%. Interestingly, this feature-level fusion outperformed the FUSED model, which relied on raw geometric registration. This suggests that deep learning models are more effective at reconciling multi-view data in feature space than traditional algorithms like ICP are at aligning noisy sensor data.
Furthermore, the SOLO-FEAT architecture demonstrated remarkable resilience to rotational noise. It maintained approximately 85% accuracy even when cameras were digitally rotated by 20 degrees—a scenario where other models' performance dropped to ~30%. This makes the architecture particularly suitable for multi-robot systems where camera calibration might be imperfect. Overall, the dual-camera approach reduced misclassifications by over half compared to single-camera baselines while requiring 12.8 times fewer points than similar point-cloud-based recognizers.
Conclusion
DuoDepth successfully tackles the occlusion problem in static gesture recognition through hardware redundancy and optimized deep learning architectures. While currently limited to static signs, the system's efficiency and resilience to noise provide a strong foundation for future work. Potential expansions include incorporating temporal data for dynamic gestures and testing the system on mobile or uncalibrated multi-robot platforms.
Summary updated