Neural Music Synthesis for Flexible Timbre Control

Jong Wook Kim, Rachel M. Bittner, Aparna Kumar, Juan Pablo Bello

TL;DR

This paper introduces Mel2Mel, a novel neural network architecture for synthesizing realistic musical audio with fine-grained control over timbre. By learning a continuous "timbre space" through instrument embeddings, Mel2Mel enables flexible timbre manipulation and morphing, overcoming limitations of traditional synthesis methods.

What Problem Does This Paper Address?

Creating realistic musical sounds with intuitive control over timbre has been a long-standing challenge in music synthesis. Traditional methods often rely on fixed presets or limited parameter spaces, making it difficult to achieve a wide range of timbres or smoothly transition between them. While recent advances in raw audio synthesis like WaveNet have shown promise, controlling the specific sonic characteristics (timbre) of the generated music in a flexible and expressive way remains an open problem. This paper aims to bridge this gap by developing a data-driven approach that learns timbre representations directly from audio data.

Methods

The core of the Mel2Mel system is a recurrent neural network (RNN) architecture that takes a symbolic music representation (piano roll of notes and velocities) and a learned instrument embedding as input to predict a Mel spectrogram. This Mel spectrogram is then fed into a WaveNet vocoder to generate the final audio.

Key novelties include:

  • Learned Instrument Embeddings: Instead of relying on predefined instrument categories, Mel2Mel learns a continuous, multi-dimensional embedding space for instruments. This space captures nuanced timbral variations and allows for interpolation between instruments.
  • FiLM (Feature-wise Linear Modulation) Layers: These layers are crucial for timbre conditioning. They use the learned instrument embedding to dynamically transform the intermediate features of the RNN, effectively shaping the sound's spectral and temporal envelopes according to the desired timbre.
  • Combined Onset and Frame Input: The model uses both note onset and frame information from the piano roll, which has been shown to improve synthesis quality by capturing both the attack transients and sustained parts of notes.
  • WaveNet Vocoder: A standard WaveNet model is used for the final audio synthesis, leveraging its proven ability to generate high-fidelity audio from Mel spectrograms.

Datasets

The authors used synthesized audio for training and evaluation. They started with 10 General MIDI instruments and 334 piano tracks synthesized using FluidSynth with the default SoundFont from MuseScore 3. These tracks were derived from MIDI files found on www.piano-midi.de. For later experiments, they also generated a dataset using 100 manually selected instrument classes with high-quality SoundFonts. The total synthesized dataset comprised 3,340 tracks and 221 hours of audio.

Key Insights and Ideas

  • Timbre as a Continuous Space: The most significant insight is that timbre can be effectively represented and manipulated within a continuous, learned embedding space. This allows for smooth timbre morphing, a capability largely absent in traditional synthesis.
  • FiLM Layers for Timbre Control: The FiLM layers are surprisingly effective at translating abstract timbre embeddings into concrete sonic characteristics by modulating the network's internal representations.
  • Optimal Architecture for Generalization: The ablation study revealed that adding complexity (e.g., more layers, ReLU activations) led to overfitting, suggesting that the proposed architecture with nonlinearity primarily in the recurrent layer is optimal for generalizing timbre features.
  • Trade-off in Mel Spectrogram Representation: The paper highlights that an 80-bin Mel spectrogram is a sweet spot for WaveNet conditioning, balancing accuracy with computational feasibility.

Key Results

  • Flexible Timbre Control and Morphing: The model successfully generates audio with controllable timbres and demonstrates timbre morphing by interpolating within the learned embedding space.
  • High Synthesis Quality: Numerical evaluations (Pearson correlations) and subjective listening tests (Mean Opinion Scores - MOS) showed that the Mel2Mel model produces high-quality audio, with MOS scores approaching those of μ-law encoded/decoded audio and significantly outperforming simpler WaveNet configurations. The tanh-log-abs MSE loss function yielded the best subjective quality.
  • Optimal Architecture: Ablation studies confirmed that the proposed architecture, using both onset and frame inputs and two FiLM layers, performed best. Increasing model capacity led to overfitting.

Limitations and Future Work

  • Dataset Limitations: The current dataset is synthesized, and the authors acknowledge that training on real audio would be ideal.
  • WaveNet Vocoder Constraints: The use of nv-wavenet, an open-source implementation, imposed limitations on channel size and layers, potentially affecting synthesis quality. Future work could explore Parallel WaveNet for faster and potentially higher-quality synthesis.
  • Quantization Noise: The μ-law encoding introduces quantization noise, which is an inherent limitation of the WaveNet vocoder.
  • Real Audio Synthesis: The authors suggest extending the model to be trained on real audio data for even more realistic and diverse synthesis.
  • Improved Vocoders and Loss Functions: Investigating alternative vocoders and adversarial loss functions for Mel spectrogram prediction could further enhance synthesis accuracy.
  • Exploring Higher-Dimensional Embeddings: While a 10-dimensional embedding was explored, further research into higher dimensions could capture even more complex timbral nuances.

Summary updated

Abstract

The recent success of raw audio waveform synthesis models like WaveNet motivates a new approach for music synthesis, in which the entire process --- creating audio samples from a score and instrument information --- is modeled using generative neural networks. This paper describes a neural music synthesis model with flexible timbre controls, which consists of a recurrent neural network conditioned on a learned instrument embedding followed by a WaveNet vocoder. The learned embedding space successfully captures the diverse variations in timbres within a large dataset and enables timbre control and morphing by interpolating between instruments in the embedding space. The synthesis quality is evaluated both numerically and perceptually, and an interactive web demo is presented.
BibTeX
@inproceedings{kim2019neural,
  author = {Jong Wook Kim and Rachel M. Bittner and Aparna Kumar and Juan Pablo Bello},
  title = {{Neural Music Synthesis for Flexible Timbre Control}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  doi = {10.1109/icassp.2019.8683596},
}
Figure