Marker-removal Networks to Collect Precise 3D Hand Data for RGB-based Estimation and its Application in Piano

Erwin Wu, Hayato Nishioka, Shinichi Furuya, Hideki Koike

Abstract

Accurate 3D hand pose estimation is a critical component for human-computer interaction, yet it remains a significant challenge for high-level dexterous skills like piano playing. Traditional marker-based motion capture (MoCap) systems provide high precision but introduce visual artifacts that hinder the training of standard RGB camera models. This paper introduces a novel "Marker-removal Network" (MR-Net) that bridges the gap between precise MoCap data and realistic bare-hand imagery. By leveraging a massive new dataset, PianoHand2.5M, and a specialized estimation network called PiaNet, we demonstrate state-of-the-art performance in tracking complex, highly occluded hand movements characteristic of professional piano performance.

Introduction

3D hand pose estimation is often considered a "holy grail" for human-computer interaction. However, current solutions typically fall into two categories, both of which have significant drawbacks for specialized tasks. Marker-based systems are incredibly accurate but require bulky reflective markers that distract performers and create unnatural visual features. Conversely, markerless RGB systems rely on deep learning but require massive, accurately labeled datasets. Manual labeling is labor-intensive, and synthetic (CGI) data often suffers from a "domain gap," failing to capture the nuances of real-world lighting and skin textures.

Piano playing presents a unique challenge due to "awkward" poses—such as the "thumb-under" maneuver—and heavy self-occlusion. General-purpose hand trackers often fail to resolve these positions, necessitating a domain-specific approach.

Methods: MR-Net and PiaNet

The core innovation of this research is a two-stage pipeline designed to create a high-fidelity dataset without manual labor.

The Marker-removal Network (MR-Net)

Instead of synthesizing hands from scratch, the authors developed MR-Net to "photoshop" markers out of real MoCap footage. To train this network, they employed a clever inversion: they took images of bare hands, synthetically added markers to them (Stage I: Marker Synthetization), and then trained the network to remove them. This process results in realistic, bare-hand images that retain the sub-millimeter 3D coordinate data from the original MoCap.

Overview of the MR-Net approach, showing the pipeline from MoCap to the PianoHand dataset and qualitative comparisons against other models.

PiaNet and PiaSim

The baseline estimation network, PiaNet, utilizes a ResNet-50 backbone to predict 3D heat maps for hand joints. To further enhance robustness, the researchers integrated the PiaSim Module, an LSTM-based component that predicts keystrokes. By correlating hand motion with the physical act of pressing a key, the model can better resolve fingertip positions that might otherwise be hidden by the rest of the hand.

The first stage of the MR-Net pipeline: Marker Synthetization, where markers are added to bare-hand images to train the removal network.

The PianoHand2.5M Dataset

To support this research, the authors introduced PianoHand2.5M, a massive dataset featuring:

  • 21 experienced pianists, including professionals.
  • 2.5 million images captured at high frame rates (120-240 fps).
  • Dual Environments: Captures from both an electric piano studio (green screen) and a grand piano studio.
  • Ground Truth: High-precision 3D coordinates for 23 points per hand, synchronized with MIDI data and IR-sensor-based keystroke depth.

Key Results and Insights

The MR-Net approach offers several advantages over traditional methods. The most significant insight is that removing tiny markers from a real image is computationally easier for a Generative Adversarial Network (GAN) than synthesizing an entire realistic hand. This preserves complex skin textures and specific environmental lighting while maintaining "gold standard" labels.

Quantitative Performance

  • Superior Accuracy: The MR-Net trained model achieved a Mean Per Joint Position Error (MPJPE) of 9.22mm, significantly outperforming models trained on raw MoCap data (14.44mm) or purely synthetic data (11.12mm).
  • Outperforming State-of-the-Art: When tested on piano sequences from the CMU Panoptic dataset, PiaNet achieved a 98.1% PCK (Percentage of Correct Keypoints), surpassing other leading models like InterHand (94.1%).

Qualitative Validation

In a blind study, eight experienced pianists ranked the poses generated by PiaNet as significantly more natural and anatomically correct than those from general-purpose trackers. The system proved particularly adept at handling difficult cases like thumb-under moves and finger occlusions.

Limitations and Future Work

While highly effective, the system has limitations. MR-Net may occasionally "clean" off distinct hand features like moles or hair, mistaking them for markers. Additionally, while the dataset includes two-handed play, the current PiaNet model focuses primarily on single-hand estimation. Future research will aim to tackle the complex "hand-on-hand" occlusions common in advanced piano literature and explore the integration of multi-view camera setups to further refine accuracy.

Summary updated

Abstract

Hand pose analysis is a key step to understanding dexterous hand performances of many high-level skills, such as playing the piano. Currently, most accurate hand tracking systems are using fabric-/marker-based sensing that potentially disturbs users’ performance. On the other hand, markerless computer vision-based methods rely on a precise bare-hand dataset for training, which is difficult to obtain. In this paper, we collect a large-scale high precision 3D hand pose dataset with a small workload using a marker-removal network (MR-Net). The proposed MR-Net translates the marked-hand images to realistic bare-hand images, and the corresponding 3D postures are captured by a motion capture thus few manual annotations are required. A baseline estimation network PiaNet is introduced and we report the accuracy of various metrics together with a blind qualitative test to show the practical effect.
BibTeX
@inproceedings{wu2023marker,
  author = {Erwin Wu and Hayato Nishioka and Shinichi Furuya and Hideki Koike},
  title = {{Marker-removal Networks to Collect Precise 3D Hand Data for RGB-based Estimation and its Application in Piano}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00299},
}
Figure