RP1M: A Large-Scale Motion Dataset for Piano Playing with Bi-Manual Dexterous Robot Hands

Yi Zhao, Le Chen, Jan Schneider, Quankai Gao, Juho Kannala, Bernhard Schölkopf, Joni Pajarinen, Dieter Büchler

Introduction

Endowing robotic hands with human-level dexterity remains a significant challenge in the field of robotics. Piano playing represents a unique intersection of high-speed coordination, precise force application, and the need for generalization across vast musical repertoires. Historically, the primary bottleneck has been the lack of large-scale, annotated datasets. Traditional methods rely on human fingering annotations, which are expensive to produce and often do not translate well to robot hand morphologies.

Methodology: Automatic Fingering via Optimal Transport

To overcome the data scarcity, researchers developed RP1M, the first large-scale motion dataset for bimanual dexterous piano playing. Central to this work is a novel approach to finger placement formulated as an Optimal Transport (OT) problem. Instead of imitating human labels, the system determines the most efficient finger-to-key mapping by minimizing the total distance fingers must travel to hit the required notes.

Overview of the RP1M data collection pipeline and the Optimal Transport-based fingering cost matrix.

This OT-based reward system allows robots to "discover" efficient fingering strategies autonomously. The researchers then trained individual Reinforcement Learning (RL) specialists for approximately 2,000 songs using the DroQ algorithm, generating over one million expert trajectories.

The RP1M Dataset

The resulting RP1M dataset contains ~1 million trajectories covering a diverse range of music sourced from the PIG and GiantMIDI-Piano datasets. Unlike previous benchmarks focused on static grasping or general manipulation, RP1M is specifically tailored for high-speed, dynamic, bimanual tasks. This scale enables the benchmarking of "Generalist" models, such as Behavior Transformers (BeT) and Diffusion Policies (DP), to evaluate zero-shot performance on unseen musical pieces.

Key Results and Insights

The RL specialist agents demonstrated high-fidelity playing, with 79% achieving an F1 score above 0.75. The OT-based strategy proved particularly effective on complex pieces like French Suite No. 5, where the robot outperformed human-mimicking strategies by finding robot-optimal paths that human hands might find physically impossible.

Learning curves showing F1 scores across different musical pieces for the OT-based agent compared to baselines.

Furthermore, the study revealed a clear scaling law: as the training set size increased, the zero-shot performance of imitation learning models improved steadily. The Transformer-based Diffusion Policy (DP-T) emerged as the strongest generalist, showing significant gains in F1 scores as the dataset grew from 12 to 150 songs.

Conclusion

RP1M represents a major step forward in multi-task robotic dexterity. By moving from human imitation to mathematical discovery via Optimal Transport, the researchers have created a scalable framework for teaching robots complex, high-speed tasks. Future work will focus on closing the gap between specialist and generalist models and incorporating multi-modal feedback—such as vision and audio—to achieve true human-level performance on physical hardware.

Summary updated

BibTeX
@inproceedings{zhao2024rp,
  author = {Yi Zhao and Le Chen and Jan Schneider and Quankai Gao and Juho Kannala and Bernhard Schölkopf and Joni Pajarinen and Dieter Büchler},
  title = {{RP1M: A Large-Scale Motion Dataset for Piano Playing with Bi-Manual Dexterous Robot Hands}},
  booktitle = {Conference on Robot Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=Y7yLmkSWI6},
}
Figure