Abstract
Accurate 3D hand pose estimation is a critical component for human-computer interaction, yet it remains a significant challenge for high-level dexterous skills like piano playing. Traditional marker-based motion capture (MoCap) systems provide high precision but introduce visual artifacts that hinder the training of standard RGB camera models. This paper introduces a novel "Marker-removal Network" (MR-Net) that bridges the gap between precise MoCap data and realistic bare-hand imagery. By leveraging a massive new dataset, PianoHand2.5M, and a specialized estimation network called PiaNet, we demonstrate state-of-the-art performance in tracking complex, highly occluded hand movements characteristic of professional piano performance.
Introduction
3D hand pose estimation is often considered a "holy grail" for human-computer interaction. However, current solutions typically fall into two categories, both of which have significant drawbacks for specialized tasks. Marker-based systems are incredibly accurate but require bulky reflective markers that distract performers and create unnatural visual features. Conversely, markerless RGB systems rely on deep learning but require massive, accurately labeled datasets. Manual labeling is labor-intensive, and synthetic (CGI) data often suffers from a "domain gap," failing to capture the nuances of real-world lighting and skin textures.
Piano playing presents a unique challenge due to "awkward" poses—such as the "thumb-under" maneuver—and heavy self-occlusion. General-purpose hand trackers often fail to resolve these positions, necessitating a domain-specific approach.
Methods: MR-Net and PiaNet
The core innovation of this research is a two-stage pipeline designed to create a high-fidelity dataset without manual labor.
The Marker-removal Network (MR-Net)
Instead of synthesizing hands from scratch, the authors developed MR-Net to "photoshop" markers out of real MoCap footage. To train this network, they employed a clever inversion: they took images of bare hands, synthetically added markers to them (Stage I: Marker Synthetization), and then trained the network to remove them. This process results in realistic, bare-hand images that retain the sub-millimeter 3D coordinate data from the original MoCap.
PiaNet and PiaSim
The baseline estimation network, PiaNet, utilizes a ResNet-50 backbone to predict 3D heat maps for hand joints. To further enhance robustness, the researchers integrated the PiaSim Module, an LSTM-based component that predicts keystrokes. By correlating hand motion with the physical act of pressing a key, the model can better resolve fingertip positions that might otherwise be hidden by the rest of the hand.
The PianoHand2.5M Dataset
To support this research, the authors introduced PianoHand2.5M, a massive dataset featuring:
- 21 experienced pianists, including professionals.
- 2.5 million images captured at high frame rates (120-240 fps).
- Dual Environments: Captures from both an electric piano studio (green screen) and a grand piano studio.
- Ground Truth: High-precision 3D coordinates for 23 points per hand, synchronized with MIDI data and IR-sensor-based keystroke depth.
Key Results and Insights
The MR-Net approach offers several advantages over traditional methods. The most significant insight is that removing tiny markers from a real image is computationally easier for a Generative Adversarial Network (GAN) than synthesizing an entire realistic hand. This preserves complex skin textures and specific environmental lighting while maintaining "gold standard" labels.
Quantitative Performance
- Superior Accuracy: The MR-Net trained model achieved a Mean Per Joint Position Error (MPJPE) of 9.22mm, significantly outperforming models trained on raw MoCap data (14.44mm) or purely synthetic data (11.12mm).
- Outperforming State-of-the-Art: When tested on piano sequences from the CMU Panoptic dataset, PiaNet achieved a 98.1% PCK (Percentage of Correct Keypoints), surpassing other leading models like InterHand (94.1%).
Qualitative Validation
In a blind study, eight experienced pianists ranked the poses generated by PiaNet as significantly more natural and anatomically correct than those from general-purpose trackers. The system proved particularly adept at handling difficult cases like thumb-under moves and finger occlusions.
Limitations and Future Work
While highly effective, the system has limitations. MR-Net may occasionally "clean" off distinct hand features like moles or hair, mistaking them for markers. Additionally, while the dataset includes two-handed play, the current PiaNet model focuses primarily on single-hand estimation. Future research will aim to tackle the complex "hand-on-hand" occlusions common in advanced piano literature and explore the integration of multi-view camera setups to further refine accuracy.
Summary updated