Self-Calibrated Multi-Sensor Wearable for Hand Tracking and Modeling

Nikhil Gosala, Fangjinhua Wang, Zhaopeng Cui, Hanxue Liang, Oliver Glauser, Shihao Wu, Olga Sorkine-HornungIEEE TVCG 2023 · DOI · OpenReview

Introduction

Hand tracking is a fundamental component of virtual and augmented reality (VR/AR) and human-computer interaction. However, current solutions often face a trade-off between accuracy and robustness. Vision-based systems, such as RGB-D cameras, provide high precision but fail when the hand leaves the camera's field of view (FoV) or is obscured by objects. Conversely, wearable systems like data gloves and inertial measurement units (IMUs) offer occlusion resistance but typically suffer from sensor drift and require tedious, user-specific calibration due to variations in hand anatomy and skin-stretch characteristics.

This paper presents a hybrid hand-tracking system that fuses an RGB-D camera with a stretch-sensing soft glove and three IMUs. The system provides consistent 3D pose estimation even under total occlusion through an automated "online" calibration process. This process leverages high-confidence camera data to personalize the wearable sensors' models in real-time, eliminating the need for manual setup.

Methods

The system architecture consists of two primary modules: Local Hand Modeling for finger poses and Global Hand Tracking for wrist positioning.

Local Hand Modeling and Side-Tuning

To handle finger poses, the system utilizes a deep network (DenseReg) to generate 2D and 3D joint heatmaps from camera data. The system calculates a confidence score ($\alpha$) by analyzing heatmap entropy (cluster count, variance, and local maxima). When the camera's confidence is low—due to motion blur or partial occlusion—the system automatically weights the glove data more heavily.

The wearable system maps physical hand poses to a virtual model during object manipulation.

To personalize the glove model without the instability of standard fine-tuning, the authors employ "side-tuning." This involves keeping a base "generic" glove model fixed while training a lightweight "side" network that learns the specific quirks of the user's hand. This calibration occurs in the background whenever the camera has a clear view of the hand.

Global Tracking and Auto-Calibration

For global tracking, an Extended Kalman Filter (EKF) fuses IMU data with camera observations. The system automatically estimates the user's arm segment lengths (upper arm and forearm) by comparing IMU rotations to the camera's observed wrist positions over time. The hardware stack includes a stretch-sensing soft glove, three Bosch BNO055 IMUs (placed on the head, upper arm, and forearm), and an Intel RealSense RGB-D camera.

Key Results

The hybrid approach demonstrates significant improvements over single-modality systems. Global hand position accuracy is approximately 7% higher than camera-only methods and 11% higher than IMU-only methods. Local hand pose precision (finger angles) improved by 11% over camera-only and 15% over glove-only setups.

Occlusion Robustness

A primary strength of the system is its resilience to occlusion. Qualitative evaluations show the system maintaining realistic hand poses even when the hand is completely hidden behind objects or moved behind the user's back.

The system maintains tracking even when the hand is occluded by a piece of paper.

Calibration Performance

The auto-calibration algorithm for skeleton lengths achieved an average error of only 5.77 cm during camera occlusion, compared to 15.68 cm for a manually calibrated model. This suggests the automated system is more effective at measuring user-specific dimensions than manual measurement.

Discussion and Future Work

While the system is robust, it currently assumes a static torso. If a user moves their chest or shoulders, global tracking error increases. Future iterations could address this by adding a fourth IMU to the trunk. Additionally, while the camera provides a confidence metric, the glove model does not yet report its own uncertainty (e.g., during sensor over-stretch). Expanding this self-calibrating logic to a full-body suit represents a promising direction for high-fidelity motion capture in unconstrained environments.

Summary updated

Abstract

We present a multi-sensor system for consistent 3D hand pose tracking and modeling that leverages the advantages of both wearable and optical sensors. Specifically, we employ a stretch-sensing soft glove and three IMUs in combination with an RGB-D camera. Different sensor modalities are fused based on the availability and confidence estimation, enabling seamless hand tracking in challenging environments with partial or even complete occlusion. To maximize the accuracy while maintaining high ease-of-use, we propose an automated user calibration that uses the RGB-D camera data to refine both the glove mapping model and the multi-IMU system parameters. Extensive experiments show that our setup outperforms the wearable-only approaches when the hand is in the field-of-view and outplays the camera-only methods when the hand is occluded.
BibTeX
@article{gosala2023self,
  author = {Nikhil Gosala and Fangjinhua Wang and Zhaopeng Cui and Hanxue Liang and Oliver Glauser and Shihao Wu and Olga Sorkine-Hornung},
  title = {{Self-Calibrated Multi-Sensor Wearable for Hand Tracking and Modeling}},
  journal = {IEEE Trans. Vis. Comput. Graph.},
  year = {2023},
  doi = {10.1109/tvcg.2021.3131230},
}
Figure