ShaRPy: Shape Reconstruction and Hand Pose Estimation from RGB-D with Uncertainty

Vanessa Wirth, Anna-Maria Liphardt, Birte Coppers, Johanna Bräunig, Simon Heinrich, Arnd Kleyer, Georg Schett, Martin Vossiek, Bernhard Egger, Marc Stamminger

TL;DR

ShaRPy is a novel markerless hand-tracking system that uses a single consumer-grade RGB-D camera to reconstruct metrically accurate 3D hand poses and personalized shapes. Its standout feature is an integrated uncertainty estimation that flags unobserved or error-prone regions, providing a "reliability guardrail" essential for clinical applications like monitoring inflammatory musculoskeletal diseases.

What Problem Does This Paper Address?

In clinical settings—such as monitoring Rheumatoid Arthritis (RA)—doctors need more than just a "plausible-looking" hand pose; they need precise, interpretable, and reliable measurements of joint angles. While marker-based motion capture is the gold standard, it is expensive and cumbersome.

Existing markerless computer vision methods often struggle with two things:

  1. Metric Accuracy: Many RGB-only methods can't determine the exact scale or depth of a hand, making them unsuitable for medical measurements.
  2. Hallucination: Standard keypoint detectors (like OpenPose) often "guess" the position of hidden fingers or mislabel joints during occlusions without warning the user.

ShaRPy addresses these gaps by combining deep learning with traditional geometric optimization to provide both a "digital twin" of the patient's hand and a clear indication of when the system's output shouldn't be trusted.

Methods

ShaRPy employs a hybrid pipeline that bridges data-driven neural networks and physics-based energy minimization:

  • Dense Correspondence Space: The authors introduce a novel way to represent the hand surface using a continuous Hue-Saturation-Value (HSV) color cylinder. A neural network (based on Yolact) is trained to predict these "semantic colors" for every pixel. This allows the system to map any pixel in the 2D image directly to a specific point on a 3D hand model.
  • Anatomically Constrained MANO Model: The system uses the popular MANO parametric hand model but modifies it to enforce medical realism. They reduced the degrees of freedom (DoF) from 45 to 23, ensuring joints only move within biologically possible ranges (e.g., preventing fingers from bending backward).
  • Two-Stage Optimization: First, the system performs a coarse initialization using the network's predictions. Then, it runs a refinement stage that minimizes an "energy" function, aligning the 3D model with both the predicted semantic features and the actual depth data from the camera.
  • Uncertainty Estimation: This is the core innovation. By comparing the final optimized model against the initial network predictions and the raw depth map, ShaRPy identifies regions that are either unobserved (hidden) or inconsistent (error-prone). These are visualized in red on the hand surface.
Figure 2. Overview of ShaRPy. First, a network based on Yolact [2] detects hands and regresses features in a correspondence space. The outputs and the depth map are used in a subsequent energy minimization framework for pose and shape estimation. Lastly, we detect uncertainties with respect to the pose parameters. As the network weights are learned, the first part of the pipeline is fixed at inference time. However, the remaining parts are still adaptable at inference time, i.e., the weights can be individually tuned for different hand scenes in clinical setups to improve performance.

Datasets

The model was trained and evaluated using several major hand-tracking datasets:

  • Training: HO3D, InterHand2.6M, H2O-3D, and FreiHAND.
  • Evaluation: The H2O dataset, which features egocentric views of hands manipulating objects—a scenario very similar to clinical hand-function tests.
  • Clinical Validation: The authors also recorded a new sequence involving a 61-year-old patient with Psoriatic Arthritis to demonstrate real-world applicability.

Key Insights and Ideas

  • Continuous vs. Discrete: By using a continuous HSV space for correspondences, the network can learn fine-grained surface details while still allowing for easy discretization into 20 distinct hand segments (fingers, phalanges, etc.).
  • Personalized Shape Matters: Unlike keypoint-only methods, ShaRPy estimates the "thickness" and overall shape of the hand. This is crucial because a "neutral" pose for a thick hand looks very different from a "neutral" pose for a thin hand in 3D space.
  • Filtering for Reliability: In clinical tasks like the "rotating ball" test, the authors show that simply ignoring frames where uncertainty is high leads to much more stable and medically useful data.
Figure 1. Compared to keypoint approaches, e.g. OpenPose [3, 26], ShaRPy estimates the 3D hand pose and shape, and indicates uncertainty by detecting unobserved and error-prone regions (both visualized in red on the hand surface).

Key Results

  • Benchmark Performance: On the H2O public leaderboard, ShaRPy ranked 1st or 3rd (depending on the version) in Mean End-point Error (MEPE), achieving an accuracy of approximately 19-20mm even though it wasn't specifically tuned for that challenge.
  • Clinical Stability: During a wrist rotation task, filtering out "uncertain" segments reduced the variance in measured joint angles significantly—from a noisy 112.55° down to a stable 18.16° for the middle finger.
  • Qualitative Superiority: As shown in Figure 1, while standard tools like OpenPose might mislabel an occluded finger, ShaRPy recognizes the occlusion and flags it as "detected uncertainty."

Limitations and Future Work

  • Generalization: While the "V3" version of the model showed strong zero-shot performance on new datasets, the system is still somewhat dependent on the diversity of its training data.
  • Depth Quality: The system relies on consumer-level RGB-D sensors (like the Azure Kinect or RealSense). Extremely noisy depth data in very bright or very dark environments could still impact accuracy.
  • Future Directions: The authors suggest integrating more complex biomedical simulations and exploring how object interaction (e.g., squeezing a ball) can be used as a more sophisticated biomarker for disease activity.

Summary updated

Abstract

Despite their potential, markerless hand tracking technologies are not yet applied in practice to the diagnosis or monitoring of the activity in inflammatory musculoskeletal diseases. One reason is that the focus of most methods lies in the reconstruction of coarse, plausible poses, whereas in the clinical context, accurate, interpretable, and reliable results are required. Therefore, we propose ShaRPy, the first RGB-D Shape Reconstruction and hand Pose tracking system, which provides uncertainty estimates of the computed pose, e.g., when a finger is hidden or its estimate is inconsistent with the observations in the input, to guide clinical decision-making. Besides pose, ShaRPy approximates a personalized hand shape, promoting a more realistic and intuitive understanding of its digital twin. Our method requires only a light-weight setup with a single consumer-level RGB-D camera yet it is able to distinguish similar poses with only small joint angle deviations in a metrically accurate space. This is achieved by combining a data-driven dense correspondence predictor with traditional energy minimization. To bridge the gap between interactive visualization and biomedical simulation we leverage a parametric hand model in which we incorporate biomedical constraints and optimize for both, its pose and hand shape. We evaluate ShaRPy on a keypoint detection benchmark and show qualitative results of hand function assessments for activity monitoring of musculoskeletal diseases.
BibTeX
@misc{wirth2023sharpy,
  author = {Vanessa Wirth and Anna-Maria Liphardt and Birte Coppers and Johanna Bräunig and Simon Heinrich and Arnd Kleyer and Georg Schett and Martin Vossiek and Bernhard Egger and Marc Stamminger},
  title = {{ShaRPy: Shape Reconstruction and Hand Pose Estimation from RGB-D with Uncertainty}},
  year = {2023},
  eprint = {2303.10042},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.10042},
}
Figure