Demonstration of PerformanceNet: A Convolutional Neural Network Model for Score-to-Audio Music Generation

Yu-Hua Chen, Bryan Wang, Yi-Hsuan Yang

Abstract

We present in this paper PerformanceNet, a neural network model we proposed recently to achieve score-to-audio music generation. The model learns to convert a music piece from the symbolic domain to the audio domain, assigning performance-level attributes such as changes in velocity automatically to the music and then synthesizing the audio. The model is therefore not just a neural audio synthesizer, but an AI performer that learns to interpret a musical score in its own way. The code and sample outputs of the model can be found online at https://github.com/bwang514/PerformanceNet.

Introduction

PerformanceNet is a convolutional neural network (CNN) designed to act as an "AI performer," transforming symbolic musical scores (such as MIDI) directly into expressive audio recordings. Unlike traditional synthesizers that generate isolated notes, this model learns to incorporate human-like nuances—such as vibrato, dynamics, and smooth transitions—across entire musical phrases. By treating music generation as a domain translation task, PerformanceNet bridges the gap between robotic MIDI playback and high-fidelity, expressive human performance.

The "Missing Middle" in AI Music

In the field of AI music, research has historically been divided into two camps: symbolic generation (algorithmic composition) and audio-domain synthesis. Symbolic generation creates MIDI files that often sound mechanical when played through standard software. Conversely, audio-domain generation focuses on synthesis, often producing high-quality sounds but typically limited to isolated notes (note-level synthesis).

The gap between these two approaches lies in performance-level attributes. When a human musician reads a score, they do not simply play notes at a fixed volume. They interpret the music, adding slides between notes, varying vibrato, and adjusting dynamics based on the musical context. These "phrase-level" attributes are rarely captured in symbolic notation but are essential for music to sound authentic. PerformanceNet aims to bridge this gap by learning the mapping from a score to a fully realized, expressive audio performance.

Methodology

The model treats music generation as a domain translation problem, translating a "pianoroll" (a binary matrix of notes) into a "spectrogram" (a visual representation of audio frequencies).

Comparison of neural audio synthesis approaches and the proposed PerformanceNet architecture.

The architecture consists of two primary subnets:

  1. ContourNet: A convolutional encoder-decoder that performs the initial translation. It takes the symbolic pianoroll and generates a "rough" spectrogram, focusing on the overall shape, timing, and pitch of the music.
  2. TextureNet: To address the potential blurriness of the initial output, TextureNet acts as a refinement layer. Using multi-band residual blocks, it sharpens frequency bins and adds realistic overtones (partials), which are crucial for authentic instrument timbre.

The model is trained end-to-end to predict magnitude spectrograms, which are subsequently converted back into playable audio.

Datasets and Training

PerformanceNet is trained using pairs of symbolic representations and audio recordings of musical phrases. Unlike models trained on datasets like NSynth, which focus on isolated notes, PerformanceNet requires multi-note sequences to learn how notes connect. The training data involves recordings of instruments including flute, piano, violin, and cello, allowing the model to learn instrument-specific performance techniques like legato and vibrato.

Key Insights and Results

A significant advancement of this work is the shift from note-level to phrase-level synthesis. By training on sequences, the model learns "score-informed" techniques that are impossible to capture in isolation. Furthermore, by using CNNs on spectrograms, the researchers leverage computer vision techniques for audio, treating instrument "texture" similarly to artistic style in image processing.

The PerformanceNet graphical user interface for real-time score-to-audio generation.

The paper presents a functional Demo System with a graphical user interface. Key results include:

  • Real-time Generation: Users can upload MIDI files and select an instrument to generate expressive audio performances on-the-fly.
  • Expressive Nuance: The model successfully predicts dynamics and pitch modulations that match the musical context of the input score.
  • End-to-End Learning: The system demonstrates that complex mappings from symbolic scores to audio can be learned for various instruments without manually engineered rules.

Limitations and Future Work

Currently, the latent code in the encoder-decoder contains a mix of timbre, style, and pitch. Future work involves implementing disentanglement techniques to allow independent control over the "style" of a performer versus the "timbre" of the instrument. Additionally, the authors aim to improve audio quality by better handling phase information and exploring musician-specific style conditioning to capture unique artistic signatures.

Summary updated

Abstract

We present in this paper PerformacnceNet, a neural network model we proposed recently to achieve score-to-audio music generation. The model learns to convert a music piece from the symbolic domain to the audio domain, assigning performance-level attributes such as changes in velocity automatically to the music and then synthesizing the audio. The model is therefore not just a neural audio synthesizer, but an AI performer that learns to interpret a musical score in its own way. The code and sample outputs of the model can be found online at https://github.com/bwang514/PerformanceNet.
BibTeX
@inproceedings{chen2019demonstration,
  author = {Yu-Hua Chen and Bryan Wang and Yi-Hsuan Yang},
  title = {{Demonstration of PerformanceNet: A Convolutional Neural Network Model for Score-to-Audio Music Generation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=gyPitAyyFH},
}
Figure