PerformanceNet: Score-to-Audio Music Generation with Multi-Band Convolutional Residual Network

Bryan Wang, Yi-Hsuan Yang

Abstract

Music creation is typically composed of two parts: composing the musical score, and then performing the score with instruments to make sounds. While recent work has made much progress in automatic music generation in the symbolic domain, few attempts have been made to build an AI model that can render realistic music audio from musical scores. Directly synthesizing audio with sound sample libraries often leads to mechanical and deadpan results, since musical scores do not contain performance-level information, such as subtle changes in timing and dynamics. Moreover, while the task may sound like a text-to-speech synthesis problem, there are fundamental differences since music audio has rich polyphonic sounds. To build such an AI performer, we propose in this paper a deep convolutional model that learns in an end-to-end manner the score-to-audio mapping between a symbolic representation of music called the piano rolls and an audio representation of music called the spectrograms. The model consists of two subnets: the ContourNet, which uses a U-Net structure to learn the correspondence between piano rolls and spectrograms and to give an initial result; and the TextureNet, which further uses a multi-band residual network to refine the result by adding the spectral texture of overtones and timbre. We train the model to generate music clips of the violin, cello, and flute, with a dataset of moderate size. We also present the result of a user study that shows our model achieves higher mean opinion score (MOS) in naturalness and emotional expressivity than a WaveNet-based model and two commercial sound libraries.

Introduction

While AI has become proficient at composing music in symbolic formats like MIDI, "rendering" that music into realistic, expressive audio remains a significant challenge. Traditional synthesis methods rely on sample libraries that often sound "mechanical" because they lack the subtle nuances—expressive timing, dynamic shifts, and complex timbral changes—that a human musician provides. The authors identify that music synthesis is fundamentally different from text-to-speech (TTS); while speech is monophonic, music is often polyphonic with a rich harmonic structure.

PerformanceNet addresses this by acting as an "AI Performer." It learns an end-to-end mapping from a symbolic score (pianoroll) to a high-resolution audio spectrogram, bypassing the need for manual parameter tuning or deadpan sample-based synthesis.

The PerformanceNet framework distinguishes between the AI Composer and the AI Performer.

Methods

The authors treat score-to-audio generation as a combination of image-to-image translation and image super-resolution. The architecture consists of two primary sub-networks:

  1. ContourNet: A U-Net-based encoder-decoder with skip connections. It takes a sparse, binary pianoroll and generates an initial "rough" spectrogram. As the input (128 pitches) is much smaller than the output (1,025 frequency bins), the U-Net is asymmetric, gradually increasing the frequency dimension.
  2. TextureNet: A progressive multi-band residual network that refines the initial output by adding "spectral texture"—the overtones and timbre that characterize real instruments. It divides the spectrogram into increasingly finer frequency bands and applies residual learning to each.
  3. Onset/Offset Encoder: A dedicated encoder provides explicit information about note starts and ends to prevent "legato" blurring.
Visualization of the mapping from symbolic pitch (pianoroll) to audio frequency (spectrogram).

The final spectrogram is converted back into a playable audio waveform using the Griffin-Lim algorithm to estimate phase.

Key Insights

  • Separation of Concerns: By distinguishing between the "AI Composer" and "AI Performer," the model can focus purely on musical expression and timbre.
  • Frequency-Axis Progression: Unlike standard image models that grow in both dimensions, TextureNet only increases resolution along the frequency axis, as temporal structure is already established by ContourNet.
  • Multi-Band Processing: Splitting the spectrogram into bands allows the model to use specialized filters for low-frequency fundamentals versus high-frequency overtones.

Results and Evaluation

The model was trained on the MusicNet dataset, focusing on solo violin, cello, and flute. Despite a relatively small training set (approx. 87 minutes), PerformanceNet achieved impressive results:

  • Subjective Quality: In a user study with 156 participants, PerformanceNet achieved higher Mean Opinion Scores (MOS) in "naturalness" and "emotional expressivity" than professional commercial synthesizers (e.g., Logic Pro) and a WaveNet-based baseline.
  • Efficiency: The model is highly accessible; for instance, the cello model converged in just 8 hours on a single NVIDIA 1080Ti GPU.

Conclusion

PerformanceNet demonstrates that deep convolutional models can effectively learn the complex mapping between symbolic scores and expressive audio. While current limitations include audio fidelity bottlenecks from the Griffin-Lim algorithm and a lack of learned rubato (timing variation), the "AI Performer" concept opens new doors for realistic music synthesis. Future work may involve integrating GAN-based training or neural vocoders to further enhance audio crispness.

Summary updated

Abstract

Music creation is typically composed of two parts: composing the musical score, and then performing the score with instruments to make sounds. While recent work has made much progress in automatic music generation in the symbolic domain, few attempts have been made to build an AI model that can render realistic music audio from musical scores. Directly synthesizing audio with sound sample libraries often leads to mechanical and deadpan results, since musical scores do not contain performance-level information, such as subtle changes in timing and dynamics. Moreover, while the task may sound like a text-to-speech synthesis problem, there are fundamental differences since music audio has rich polyphonic sounds. To build such an AI performer, we propose in this paper a deep convolutional model that learns in an end-to-end manner the score-to-audio mapping between a symbolic representation of music called the piano rolls and an audio representation of music called the spectrograms. The model consists of two subnets: the ContourNet, which uses a U-Net structure to learn the correspondence between piano rolls and spectrograms and to give an initial result; and the TextureNet, which further uses a multi-band residual network to refine the result by adding the spectral texture of overtones and timbre. We train the model to generate music clips of the violin, cello, and flute, with a dataset of moderate size. We also present the result of a user study that shows our model achieves higher mean opinion score (MOS) in naturalness and emotional expressivity than a WaveNet-based model and two commercial sound libraries. We open our source code at https://github.com/bwang514/PerformanceNet
BibTeX
@inproceedings{wang2019performancenet,
  author = {Bryan Wang and Yi-Hsuan Yang},
  title = {{PerformanceNet: Score-to-Audio Music Generation with Multi-Band Convolutional Residual Network}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=XIFqBQvtEH},
}
Figure