PerformanceNet: Score-to-Audio Music Generation with Multi-Band Convolutional Residual Network

Bryan Wang, Yi-Hsuan Yang

Abstract

Music creation typically involves two stages: composition and performance. While AI has advanced significantly in symbolic music generation, rendering realistic audio from scores remains challenging. Traditional synthesis often sounds mechanical because scores lack performance-level nuances like timing and dynamics. PerformanceNet addresses this by learning an end-to-end mapping from symbolic pianorolls to audio spectrograms. The model uses a two-stage architecture: ContourNet for structural mapping and TextureNet for spectral refinement. Evaluated on violin, cello, and flute, PerformanceNet outperforms traditional synthesizers and WaveNet-based models in naturalness and emotional expressivity.

Introduction

Presented at AAAI-19, PerformanceNet introduces the concept of an "AI Performer." Unlike models that focus on composing MIDI or scores, PerformanceNet tackles the complex task of rendering these scores into expressive, realistic audio. By treating the problem as a cross-domain translation task, the model learns to "perform" music with the subtle variations in timing and timbre characteristic of human musicians.

The Challenge of Expressive Synthesis

The "uncanny valley" of music synthesis stems from the fact that musical scores only specify what notes to play, not how to play them. Standard MIDI synthesizers often produce "deadpan" results. Existing solutions generally require laborious manual tuning or data-hungry generative models like WaveNet, which struggle with polyphonic music and are slow to train. PerformanceNet aims for an efficient, end-to-end "score-to-audio" mapping that captures expressive nuances automatically.

Methods: From Pianorolls to Spectrograms

The authors frame the task as a combination of image-to-image translation and image super-resolution. Since a pianoroll is significantly sparser than a spectrogram, the model must learn to generate missing spectral information.

Mapping symbolic pianorolls (left) to audio spectrograms (right).

The architecture consists of two primary subnets:

  1. ContourNet: An asymmetric U-Net that generates a coarse initial spectrogram. It includes an Onset/Offset Encoder to explicitly feed note boundaries into the bottleneck, preventing note blurring.
  2. TextureNet: A multi-band residual network that refines the output. Using Multi-Band Residual (MBR) blocks, it splits the spectrogram into frequency bands to learn specific textures for different registers, ensuring sharp overtones.

Finally, the Griffin-Lim algorithm converts the generated magnitude spectrogram back into a playable audio waveform.

Datasets and Training

The model was trained using MusicNet, focusing on solo clips for Cello (49 mins), Violin (30 mins), and Flute (8 mins). To compensate for the relatively small dataset, the researchers used heavy temporal overlapping to augment the training samples. Despite the limited data, the model achieved high-quality results, with the cello model converging in just 8 hours on a single GPU.

Key Insights: The AI Performer Framework

A central contribution of this work is the distinction between the AI Composer and the AI Performer. This modularity allows the rendering process to be decoupled from composition, potentially allowing for "style" or "emotion" conditioning in future iterations.

The AI Performer framework, which renders a score into audio, following an AI Composer stage.

Key technical breakthroughs included:

  • Frequency-Domain Progressive Growing: TextureNet's ability to refine resolution across frequency bands mimics the physics of sound.
  • Note Boundary Encoding: Explicitly modeling "attack" and "release" was crucial for achieving clarity and distinguishing between different playing techniques.

Results and Evaluation

A subjective listening test with 156 participants (including music professionals) revealed that PerformanceNet achieved higher Mean Opinion Scores (MOS) for naturalness and emotion than high-end commercial synthesizers (like Logic Pro) and WaveNet-based baselines. While it outperformed "soft" synthesizers, a gap remains compared to real human performances in terms of absolute audio fidelity.

Limitations and Future Work

Current limitations include the reliance on Griffin-Lim for phase estimation, which can be improved with modern neural vocoders. Additionally, the model is currently limited to solo tracks; extending it to full orchestral mixes involves complex challenges in instrument separation and spatial mixing. Future work will also explore larger, more diverse datasets to capture a wider range of playing techniques.

Source code is available at: https://github.com/bwang514/PerformanceNet

Summary updated

Abstract

Music creation is typically composed of two parts: composing the musical score, and then performing the score with instruments to make sounds. While recent work has made much progress in automatic music generation in the symbolic domain, few attempts have been made to build an AI model that can render realistic music audio from musical scores. Directly synthesizing audio with sound sample libraries often leads to mechanical and deadpan results, since musical scores do not contain performance-level information, such as subtle changes in timing and dynamics. Moreover, while the task may sound like a text-to-speech synthesis problem, there are fundamental differences since music audio has rich polyphonic sounds. To build such an AI performer, we propose in this paper a deep convolutional model that learns in an end-to-end manner the score-to-audio mapping between a symbolic representation of music called the pianorolls and an audio representation of music called the spectrograms. The model consists of two subnets: the ContourNet, which uses a U-Net structure to learn the correspondence between pianorolls and spectrograms and to give an initial result; and the TextureNet, which further uses a multi-band residual network to refine the result by adding the spectral texture of overtones and timbre. We train the model to generate music clips of the violin, cello, and flute, with a dataset of moderate size. We also present the result of a user study that shows our model achieves higher mean opinion score (MOS) in naturalness and emotional expressivity than a WaveNet-based model and two off-the-shelf synthesizers. We open our source code at https://github.com/bwang514/PerformanceNet
BibTeX
@inproceedings{wang2019performancenet,
  author = {Bryan Wang and Yi-Hsuan Yang},
  title = {{PerformanceNet: Score-to-Audio Music Generation with Multi-Band Convolutional Residual Network}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=XIFqBQvtEH},
}
Figure