PianoMime: Learning a Generalist, Dexterous Piano Player from Internet Demonstrations

Cheng Qian, Julen Urain, Kevin Zakka, Jan Peters

Abstract

Training robotic agents for high-dexterity tasks remains a significant challenge due to the action and embodiment gaps between human demonstrations and robotic execution. We introduce PianoMime, a framework that leverages large-scale internet demonstrations from YouTube to train a generalist piano-playing agent. Our approach involves a three-stage pipeline: data preparation from video demonstrations, reinforcement learning of song-specific expert policies, and distillation into a single hierarchical diffusion policy. Our results demonstrate that PianoMime can achieve a 56% F1 score on unseen songs, significantly advancing the state-of-the-art in robotic piano performance.

Introduction

The internet provides a vast repository of human skill demonstrations, yet translating these into robotic motor commands is difficult. In the context of piano playing, YouTube offers thousands of hours of professional performances. However, three primary hurdles exist: the Action Gap (lack of torque/force data), the Embodiment Gap (kinematic differences between human and robot hands), and the Generalization Gap (the difficulty of moving beyond specific trained pieces to arbitrary new music).

PianoMime addresses these challenges by "watching" professional performances and learning to mimic them within a simulated environment. By combining reinforcement learning for expert skills with diffusion-based policy distillation, we create an agent capable of playing arbitrary new songs with high dexterity.

Methodology

The PianoMime framework is structured into three distinct phases designed to bridge the gap from video pixels to robotic joint actions.

1. Data Preparation

The first phase involves extracting usable data from YouTube. We focus on channels providing top-down views of piano performances accompanied by MIDI files. Using MediaPipe, we track human fingertip trajectories in 2D. These coordinates are then mapped into a 3D Mujoco simulation environment using a homography matrix, providing a spatial reference for the robot's target movements.

2. Expert Policy Learning

For each song in the training set, we train a "specialist" expert policy using Proximal Policy Optimization (PPO). To improve efficiency, we employ Residual Policy Learning. An Inverse Kinematics (IK) solver provides a "nominal" behavior—positioning the fingers near the correct keys—while the RL agent learns the "residual" adjustments necessary to execute the precise key presses. A style-mimicking reward is incorporated to ensure the robot's movements remain human-like and fluid.

3. Generalist Policy Distillation

To create a single agent capable of playing any song, we distill the knowledge from individual experts into a Hierarchical Diffusion Policy:

  • Goal Representation: We utilize a Signed Distance Field (SDF) autoencoder to represent the 88-key piano state. Unlike simple bit-vectors, the SDF provides a spatially-aware latent code, helping the model understand the physical proximity of keys.
  • Hierarchical Architecture: The system is split into a high-level policy that predicts desired fingertip trajectories and a low-level "inverse dynamics" model that translates these trajectories into joint-space actions.

Experimental Setup

We evaluated PianoMime using several datasets:

  • PianoX Dataset: A custom collection of 60 songs (431 clips) sourced from YouTube, totaling approximately 258,000 state-action pairs.
  • GiantMIDI: Used for pre-training the SDF goal representation encoder to ensure broad coverage of musical structures.
  • Etude-12: A benchmark of classical pieces used to evaluate the agent's ability to generalize to out-of-distribution musical styles.

Results and Analysis

Our experiments show that PianoMime significantly outperforms existing baselines. The song-specific specialist agents achieved an average F1 score of 0.94, compared to 0.75 for the RoboPianist baseline.

More importantly, the distilled generalist agent achieved a 56% F1 score on entirely unseen songs. Ablation studies confirmed that both the hierarchical design and the SDF-based goal representation were essential for this generalization; removing either component resulted in a sharp decline in performance on test songs.

We also observed that decoupling motion from the task was vital. While human motion serves as a useful "style guide," prioritizing the task reward (correct key presses) over perfect motion tracking led to superior musical performance. This suggests that the model learns the intent of the human performer rather than just copying visual artifacts.

Limitations and Future Work

Despite its success, PianoMime faces several limitations. The model currently runs at 15Hz on an RTX 4090, which is slightly below the threshold for real-time high-speed robotics. Furthermore, while the agent generalizes well to modern songs, its performance on the Etude-12 classical dataset was lower, indicating a sensitivity to musical style. Future work will focus on increasing training diversity and improving the acoustic quality of the performance, specifically regarding strike velocity and pedaling.

Summary updated

Abstract

In this work, we introduce PianoMime, a framework for training a piano-playing agent using internet demonstrations. The internet is a promising source of large-scale demonstrations for training our robot agents. In particular, for the case of piano-playing, Youtube is full of videos of professional pianists playing a wide myriad of songs. In our work, we leverage these demonstrations to learn a generalist piano-playing agent capable of playing any arbitrary song. Our framework is divided into three parts: a data preparation phase to extract the informative features from the Youtube videos, a policy learning phase to train song-specific expert policies from the demonstrations and a policy distillation phase to distil the policies into a single generalist agent. We explore different policy designs to represent the agent and evaluate the influence of the amount of training data on the generalization capability of the agent to novel songs not available in the dataset. We show that we are able to learn a policy with up to 56\% F1 score on unseen songs.
BibTeX
@inproceedings{qian2024pianomime,
  author = {Cheng Qian and Julen Urain and Kevin Zakka and Jan Peters},
  title = {{PianoMime: Learning a Generalist, Dexterous Piano Player from Internet Demonstrations}},
  booktitle = {DM @ RSS},
  year = {2024},
  url = {https://openreview.net/forum?id=mZGi8ANPxz},
}
Figure