Abstract
We present PerformanceNet, a convolutional neural network model designed for score-to-audio music generation. The model learns to convert musical pieces from the symbolic domain (e.g., MIDI) to the audio domain, automatically assigning performance-level attributes such as changes in velocity and articulation. Rather than acting as a simple note-level synthesizer, PerformanceNet functions as an "AI performer" that interprets a musical score to produce expressive audio. The code and sample outputs are available at https://github.com/bwang514/PerformanceNet.
Introduction: Bridging the Performance Gap
In AI music research, a significant divide exists between symbolic generation (composing melodies) and audio synthesis (generating sound). The missing link is the human element: performance.
When a musician reads a score, they do not play notes in isolation at a static volume. They interpret phrases, adding nuances like vibrato, legato transitions, and dynamic volume shifts. Traditional synthesizers often struggle with this because they lack the context of the musical phrase. PerformanceNet addresses this by learning a direct, end-to-end mapping from a symbolic score to a fully realized, expressive audio recording.
Methodology
PerformanceNet utilizes a two-stage convolutional architecture to handle the complexity of score-to-audio translation. Unlike standard synthesizers that process notes individually, PerformanceNet processes entire musical phrases to capture temporal context.
The architecture is composed of two primary subnets:
- ContourNet: A convolutional encoder-decoder that performs domain translation. It converts a pianoroll (a binary matrix of notes over time) into a low-resolution spectrogram, capturing the fundamental "contour" and timing of the music.
- TextureNet: This subnet acts as a refinement layer. Using multi-band residual blocks, it performs a task similar to spectral super-resolution, sharpening frequency bins and adding the complex overtones (harmonics) necessary for realistic instrument timbre.
The model is trained on pairs of symbolic scores and real-world audio recordings. This allows it to learn expressive techniques like vibrato and slides that are typically absent in MIDI-only datasets.
Key Insights and Ideas
- Phrase-level Processing: The core innovation is that synthesis should not happen in a vacuum. A note's sound is dictated by its position in a phrase and its relationship to neighboring notes.
- The "AI Performer" Paradigm: The goal shifts from building a better instrument to building a better musician. The model acts as an interpreter, assigning its own "style" to a score.
- Spectral Refinement: By splitting the task into ContourNet and TextureNet, the model mimics human perception—first identifying the melody (contour) and then the specific quality of the instrument (texture).
Results and Demonstration
The model successfully generates audio directly from MIDI without requiring hand-crafted rules for expression. TextureNet significantly improves audio quality by adding realistic overtones that standard encoder-decoders would otherwise blur.
To demonstrate the model's capabilities, the authors developed a GUI (shown above) that allows users to upload a MIDI file, select an instrument (Flute, Piano, Violin, or Cello), and generate an expressive performance on-the-fly.
Limitations and Future Work
Currently, the model's latent space mixes timbre, style, and pitch. Future work aims to disentangle these elements, which would allow a user to apply the "style" of a specific famous musician to the "timbre" of a different instrument. Additionally, while the model handles basic phrasing well, more complex techniques like guitar hammer-ons and pull-offs remain a challenge for future iterations.
Summary updated