Marker-removal Networks to Collect Precise 3D Hand Data for RGB-based Estimation and its Application in Piano

Erwin Wu, Hayato Nishioka, Shinichi Furuya, Hideki Koike

Abstract

Hand pose analysis is a key step to understanding dexterous hand performances of many high-level skills, such as playing the piano. Currently, most accurate hand tracking systems are using fabric-/marker-based sensing that potentially disturbs users’ performance. On the other hand, markerless computer vision-based methods rely on a precise bare-hand dataset for training, which is difficult to obtain. In this paper, we collect a large-scale high precision 3D hand pose dataset with a small workload using a marker-removal network (MR-Net). The proposed MR-Net translates the marked-hand images to realistic bare-hand images, and the corresponding 3D postures are captured by a motion capture thus few manual annotations are required. A baseline estimation network PiaNet is introduced and we report the accuracy of various metrics together with a blind qualitative test to show the practical effect.
BibTeX
@inproceedings{wu2023marker,
  author = {Erwin Wu and Hayato Nishioka and Shinichi Furuya and Hideki Koike},
  title = {{Marker-removal Networks to Collect Precise 3D Hand Data for RGB-based Estimation and its Application in Piano}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00299},
}

Loading related content...

Figure