Abstract
Music creation typically involves two stages: composition and performance. While AI has advanced significantly in symbolic music generation, rendering realistic audio from scores remains challenging. Traditional synthesis often sounds mechanical because scores lack performance-level nuances like timing and dynamics. PerformanceNet addresses this by learning an end-to-end mapping from symbolic pianorolls to audio spectrograms. The model uses a two-stage architecture: ContourNet for structural mapping and TextureNet for spectral refinement. Evaluated on violin, cello, and flute, PerformanceNet outperforms traditional synthesizers and WaveNet-based models in naturalness and emotional expressivity.
Introduction
Presented at AAAI-19, PerformanceNet introduces the concept of an "AI Performer." Unlike models that focus on composing MIDI or scores, PerformanceNet tackles the complex task of rendering these scores into expressive, realistic audio. By treating the problem as a cross-domain translation task, the model learns to "perform" music with the subtle variations in timing and timbre characteristic of human musicians.
The Challenge of Expressive Synthesis
The "uncanny valley" of music synthesis stems from the fact that musical scores only specify what notes to play, not how to play them. Standard MIDI synthesizers often produce "deadpan" results. Existing solutions generally require laborious manual tuning or data-hungry generative models like WaveNet, which struggle with polyphonic music and are slow to train. PerformanceNet aims for an efficient, end-to-end "score-to-audio" mapping that captures expressive nuances automatically.
Methods: From Pianorolls to Spectrograms
The authors frame the task as a combination of image-to-image translation and image super-resolution. Since a pianoroll is significantly sparser than a spectrogram, the model must learn to generate missing spectral information.
The architecture consists of two primary subnets:
- ContourNet: An asymmetric U-Net that generates a coarse initial spectrogram. It includes an Onset/Offset Encoder to explicitly feed note boundaries into the bottleneck, preventing note blurring.
- TextureNet: A multi-band residual network that refines the output. Using Multi-Band Residual (MBR) blocks, it splits the spectrogram into frequency bands to learn specific textures for different registers, ensuring sharp overtones.
Finally, the Griffin-Lim algorithm converts the generated magnitude spectrogram back into a playable audio waveform.
Datasets and Training
The model was trained using MusicNet, focusing on solo clips for Cello (49 mins), Violin (30 mins), and Flute (8 mins). To compensate for the relatively small dataset, the researchers used heavy temporal overlapping to augment the training samples. Despite the limited data, the model achieved high-quality results, with the cello model converging in just 8 hours on a single GPU.
Key Insights: The AI Performer Framework
A central contribution of this work is the distinction between the AI Composer and the AI Performer. This modularity allows the rendering process to be decoupled from composition, potentially allowing for "style" or "emotion" conditioning in future iterations.
Key technical breakthroughs included:
- Frequency-Domain Progressive Growing: TextureNet's ability to refine resolution across frequency bands mimics the physics of sound.
- Note Boundary Encoding: Explicitly modeling "attack" and "release" was crucial for achieving clarity and distinguishing between different playing techniques.
Results and Evaluation
A subjective listening test with 156 participants (including music professionals) revealed that PerformanceNet achieved higher Mean Opinion Scores (MOS) for naturalness and emotion than high-end commercial synthesizers (like Logic Pro) and WaveNet-based baselines. While it outperformed "soft" synthesizers, a gap remains compared to real human performances in terms of absolute audio fidelity.
Limitations and Future Work
Current limitations include the reliance on Griffin-Lim for phase estimation, which can be improved with modern neural vocoders. Additionally, the model is currently limited to solo tracks; extending it to full orchestral mixes involves complex challenges in instrument separation and spatial mixing. Future work will also explore larger, more diverse datasets to capture a wider range of playing techniques.
Source code is available at: https://github.com/bwang514/PerformanceNet
Summary updated