Demonstration of PerformanceNet: A Convolutional Neural Network Model for Score-to-Audio Music Generation

Yu-Hua Chen, Bryan Wang, Yi-Hsuan Yang

Abstract

We present PerformanceNet, a convolutional neural network model designed for score-to-audio music generation. The model learns to convert musical pieces from the symbolic domain (e.g., MIDI) to the audio domain, automatically assigning performance-level attributes such as changes in velocity and articulation. Rather than acting as a simple note-level synthesizer, PerformanceNet functions as an "AI performer" that interprets a musical score to produce expressive audio. The code and sample outputs are available at https://github.com/bwang514/PerformanceNet.

Introduction: Bridging the Performance Gap

In AI music research, a significant divide exists between symbolic generation (composing melodies) and audio synthesis (generating sound). The missing link is the human element: performance.

When a musician reads a score, they do not play notes in isolation at a static volume. They interpret phrases, adding nuances like vibrato, legato transitions, and dynamic volume shifts. Traditional synthesizers often struggle with this because they lack the context of the musical phrase. PerformanceNet addresses this by learning a direct, end-to-end mapping from a symbolic score to a fully realized, expressive audio recording.

Methodology

PerformanceNet utilizes a two-stage convolutional architecture to handle the complexity of score-to-audio translation. Unlike standard synthesizers that process notes individually, PerformanceNet processes entire musical phrases to capture temporal context.

Comparison between a standard note-level neural audio synthesizer (a) and the proposed PerformanceNet architecture (b).

The architecture is composed of two primary subnets:

  1. ContourNet: A convolutional encoder-decoder that performs domain translation. It converts a pianoroll (a binary matrix of notes over time) into a low-resolution spectrogram, capturing the fundamental "contour" and timing of the music.
  2. TextureNet: This subnet acts as a refinement layer. Using multi-band residual blocks, it performs a task similar to spectral super-resolution, sharpening frequency bins and adding the complex overtones (harmonics) necessary for realistic instrument timbre.

The model is trained on pairs of symbolic scores and real-world audio recordings. This allows it to learn expressive techniques like vibrato and slides that are typically absent in MIDI-only datasets.

Key Insights and Ideas

  • Phrase-level Processing: The core innovation is that synthesis should not happen in a vacuum. A note's sound is dictated by its position in a phrase and its relationship to neighboring notes.
  • The "AI Performer" Paradigm: The goal shifts from building a better instrument to building a better musician. The model acts as an interpreter, assigning its own "style" to a score.
  • Spectral Refinement: By splitting the task into ContourNet and TextureNet, the model mimics human perception—first identifying the melody (contour) and then the specific quality of the instrument (texture).

Results and Demonstration

The model successfully generates audio directly from MIDI without requiring hand-crafted rules for expression. TextureNet significantly improves audio quality by adding realistic overtones that standard encoder-decoders would otherwise blur.

The interactive PerformanceNet demo interface, allowing users to select MIDI files and instruments for generation.

To demonstrate the model's capabilities, the authors developed a GUI (shown above) that allows users to upload a MIDI file, select an instrument (Flute, Piano, Violin, or Cello), and generate an expressive performance on-the-fly.

Limitations and Future Work

Currently, the model's latent space mixes timbre, style, and pitch. Future work aims to disentangle these elements, which would allow a user to apply the "style" of a specific famous musician to the "timbre" of a different instrument. Additionally, while the model handles basic phrasing well, more complex techniques like guitar hammer-ons and pull-offs remain a challenge for future iterations.

Summary updated

Abstract

We present in this paper PerformacnceNet, a neural network model we proposed recently to achieve score-to-audio music generation. The model learns to convert a music piece from the symbolic domain to the audio domain, assigning performance-level attributes such as changes in velocity automatically to the music and then synthesizing the audio. The model is therefore not just a neural audio synthesizer, but an AI performer that learns to interpret a musical score in its own way. The code and sample outputs of the model can be found online at https://github.com/bwang514/PerformanceNet.
BibTeX
@inproceedings{chen2019demonstration,
  author = {Yu-Hua Chen and Bryan Wang and Yi-Hsuan Yang},
  title = {{Demonstration of PerformanceNet: A Convolutional Neural Network Model for Score-to-Audio Music Generation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=gyPitAyyFH},
}
Figure