Expressive Piano Performance Rendering from Unpaired Data

Lenny Renault, Rémi Mignot, Axel RoebelSPIRE 2023 · DOI · OpenAlex

Abstract

Recent advances in data-driven expressive performance rendering have enabled automatic models to reproduce the characteristics and the variability of human performances of musical compositions. However, these models need to be trained with aligned pairs of scores and performances and they rely notably on score-specific markings, which limits their scope of application. This work tackles the piano performance rendering task in a low-informed setting by only considering the score note information and without aligned data. The proposed model relies on an adversarial training where the basic score notes properties are modified in order to reproduce the expressive qualities contained in a dataset of real performances. First results for unaligned score-to-performance rendering are presented through a conducted listening test. While the interpretation quality is not on par with highly-supervised methods and human renditions, our method shows promising results for transferring realistic expressivity into scores.

Introduction

In computer music, "expressive performance rendering" aims to transform mechanical musical scores into nuanced performances by introducing human-like variations in tempo, velocity, and articulation. Traditionally, this requires labor-intensive aligned datasets where digital score notes are perfectly matched to human recordings. Furthermore, most models depend on "high-informed" scores containing detailed markings like slurs and accents. This research addresses the "low-informed" and "unpaired" challenge: training a model to play expressively using only basic MIDI note information, without requiring aligned human-score pairs.

Methods

The researchers treat expressive rendering as a domain transfer task using a Generative Adversarial Network (GAN) framework.

Overview of the proposed GAN-based architecture for expressive performance rendering.

The architecture consists of:

  1. The Generator ($G$): This model takes a "deadpan" score and predicts modifications ($\Delta X$) to adjust onset timing, note duration, and velocity. It employs a hierarchical approach, encoding notes into chords (Note-to-Chord or N2C), processing them via a bidirectional Gated Recurrent Unit (GRU), and expanding them back to notes (Chord-to-Note or C2N).
  2. The Discriminator ($D$): Three multi-scale discriminators analyze performances at different resolutions to distinguish between human recordings and generated outputs.
  3. The "Mix" Function: The model learns to perturb the original score rather than generating from scratch.
  4. Regularization ($L_{score}$): This critical loss term ensures the output remains musically faithful to the input score, preventing the GAN from altering the composition itself.

Datasets

The study utilized two major datasets: the ASAP Dataset for digital scores and the MAESTRO Dataset (v3.0.0) for human performances recorded on Yamaha Disklaviers. Because the method is unsupervised, the model was trained on the entirety of both datasets (962 performances and 107 scores) without needing explicit pairings.

Key Results and Insights

A listening test involving 19 professional musicians compared the proposed model against "Deadpan" MIDI, the highly-supervised VirtuosoNet, and real human recordings.

Comparison of listening test scores for human, VirtuosoNet, the proposed model, and deadpan performances.

Key findings include:

  • Hierarchical Modeling: Pooling notes into chords captures the vertical structure of piano music, ensuring coherent expressive treatment of simultaneous notes.
  • Performance Gains: The proposed model significantly outperformed mechanical "Deadpan" scores in naturalness, though a gap remains compared to human pianists and highly-supervised models like VirtuosoNet.
  • Stability: The $L_{score}$ regularization was essential for keeping performances grounded in the original composition.

Limitations and Future Work

The model currently struggles with slow, lyrical pieces, often applying inappropriate fast-paced expressive features. It also lacks an understanding of musical semantics, such as the difference between a climax and an introduction. Future work aims to incorporate domain labels (e.g., composer style) and integrate a differentiable synthesizer to allow learning directly from audio recordings.

Summary updated

Abstract

Recent advances in data-driven expressive performance rendering have enabled automatic models to reproduce the characteristics and the variability of human performances of musical compositions. However, these models need to be trained with aligned pairs of scores and performances and they rely notably on score-specific markings, which limits their scope of application. This work tackles the piano performance rendering task in a low-informed setting by only considering the score note information and without aligned data.<br> The proposed model relies on an adversarial training where the basic score notes properties are modified in order to reproduce the expressive qualities contained in a dataset of real performances. First results for unaligned score-to-performance rendering are presented through a conducted listening test. While the interpretation quality is not on par with highly-supervised methods and human renditions, our method shows promising results for transferring realistic expressivity into scores.
BibTeX
@inproceedings{renault2023expressive,
  author = {Lenny Renault and Rémi Mignot and Axel Roebel},
  title = {{Expressive Piano Performance Rendering from Unpaired Data}},
  booktitle = {SPIRE},
  year = {2023},
  doi = {10.5281/zenodo.8386761},
}
Figure