Introduction
Hand tracking is a fundamental component of virtual and augmented reality (VR/AR) and human-computer interaction. However, current solutions often face a trade-off between accuracy and robustness. Vision-based systems, such as RGB-D cameras, provide high precision but fail when the hand leaves the camera's field of view (FoV) or is obscured by objects. Conversely, wearable systems like data gloves and inertial measurement units (IMUs) offer occlusion resistance but typically suffer from sensor drift and require tedious, user-specific calibration due to variations in hand anatomy and skin-stretch characteristics.
This paper presents a hybrid hand-tracking system that fuses an RGB-D camera with a stretch-sensing soft glove and three IMUs. The system provides consistent 3D pose estimation even under total occlusion through an automated "online" calibration process. This process leverages high-confidence camera data to personalize the wearable sensors' models in real-time, eliminating the need for manual setup.
Methods
The system architecture consists of two primary modules: Local Hand Modeling for finger poses and Global Hand Tracking for wrist positioning.
Local Hand Modeling and Side-Tuning
To handle finger poses, the system utilizes a deep network (DenseReg) to generate 2D and 3D joint heatmaps from camera data. The system calculates a confidence score ($\alpha$) by analyzing heatmap entropy (cluster count, variance, and local maxima). When the camera's confidence is low—due to motion blur or partial occlusion—the system automatically weights the glove data more heavily.
To personalize the glove model without the instability of standard fine-tuning, the authors employ "side-tuning." This involves keeping a base "generic" glove model fixed while training a lightweight "side" network that learns the specific quirks of the user's hand. This calibration occurs in the background whenever the camera has a clear view of the hand.
Global Tracking and Auto-Calibration
For global tracking, an Extended Kalman Filter (EKF) fuses IMU data with camera observations. The system automatically estimates the user's arm segment lengths (upper arm and forearm) by comparing IMU rotations to the camera's observed wrist positions over time. The hardware stack includes a stretch-sensing soft glove, three Bosch BNO055 IMUs (placed on the head, upper arm, and forearm), and an Intel RealSense RGB-D camera.
Key Results
The hybrid approach demonstrates significant improvements over single-modality systems. Global hand position accuracy is approximately 7% higher than camera-only methods and 11% higher than IMU-only methods. Local hand pose precision (finger angles) improved by 11% over camera-only and 15% over glove-only setups.
Occlusion Robustness
A primary strength of the system is its resilience to occlusion. Qualitative evaluations show the system maintaining realistic hand poses even when the hand is completely hidden behind objects or moved behind the user's back.
Calibration Performance
The auto-calibration algorithm for skeleton lengths achieved an average error of only 5.77 cm during camera occlusion, compared to 15.68 cm for a manually calibrated model. This suggests the automated system is more effective at measuring user-specific dimensions than manual measurement.
Discussion and Future Work
While the system is robust, it currently assumes a static torso. If a user moves their chest or shoulders, global tracking error increases. Future iterations could address this by adding a fourth IMU to the trunk. Additionally, while the camera provides a confidence metric, the glove model does not yet report its own uncertainty (e.g., during sensor over-stretch). Expanding this self-calibrating logic to a full-body suit represents a promising direction for high-fidelity motion capture in unconstrained environments.
Summary updated