FlowSynth: Simplifying Complex Audio Generation Through Explorable Latent Spaces with Normalizing Flows

Philippe Esling, Naotake Masuda, Axel Chemla-Romeu-SantosIJCAI 2020 · DOI

TL;DR

This paper introduces FlowSynth, a novel system that simplifies controlling complex audio synthesizers by learning an invertible mapping between an audio's latent representation and its synthesizer parameters. This allows for intuitive exploration of sound possibilities, parameter inference from audio, and real-time gestural control, making advanced synthesis accessible to a wider audience.

What Problem Does This Paper Address?

Modern audio synthesizers, while incredibly versatile, are often hampered by a daunting number of parameters. For users without deep signal processing knowledge, understanding how parameter changes affect the output sound is challenging, leading to a steep learning curve. Existing solutions like parameter inference systems can be non-interactive, and macro-parameter controls often fail to capture the complex, non-linear relationships between synthesis parameters. This paper aims to bridge this gap by providing an intuitive, explorable, and interactive control framework for synthesizers.

Methods

The core of FlowSynth lies in formalizing synthesizer control as learning an invertible mapping between an audio's latent space and its corresponding parameter space. This is achieved using Variational Auto-Encoders (VAEs) to model the latent audio representation and normalizing flows to learn the mapping between this latent space and the synthesizer's parameter space. The system leverages these learned mappings for several key functionalities:

  • Parameter Inference: Given an audio sample, the model infers the synthesizer parameters that would produce a similar sound.
  • Latent Space Exploration: The learned latent space provides a simplified, low-dimensional representation of audio variations. This allows for intuitive exploration of sound possibilities, visualizing presets based on audio similarity, and interpolating between sounds.
  • Gestural Control: The system integrates with LeapMotion devices to map hand gestures to movements within the latent space, enabling real-time, expressive control over hundreds of synthesizer parameters.

The implementation is provided as a Max4Live interface and VST plugin, communicating with a Python server for machine learning tasks.

Datasets

The paper mentions that a dataset of synthesizer sounds and parameters for the Diva synthesizer was constructed for training the model. No new benchmarks were introduced.

Key Insights and Ideas

  • Invertible Mapping as a Unifying Framework: Using normalizing flows to create an invertible mapping between audio latent space and parameter space elegantly addresses multiple control challenges simultaneously.
  • Latent Space as an Intuitive Control Surface: The low-dimensional latent space learned by the VAE and mapped by normalizing flows acts as a powerful, semantic control surface, reducing the complexity of hundreds of parameters to a few intuitive dimensions.
  • Bridging the Gap for Non-Experts: By enabling parameter inference from audio and gestural control, FlowSynth significantly lowers the barrier to entry for using complex synthesizers, empowering both novice and expert users.
  • Unsupervised Dimensions for Semantic Control: The unsupervised nature of the learned latent dimensions allows them to capture principal components of audio variations, leading to semantically meaningful controls.

Key Results

The paper highlights that the probabilistic model developed outperforms baseline models in parameter inference, particularly for higher complexity scenarios (128 parameters). The system is designed to operate in real-time, with inference times under 20ms on a standard CPU, making it suitable for interactive use. The integration with LeapMotion allows for efficient exploration of the latent space, enabling users to control numerous parameters through simple hand movements.

Limitations and Future Work

The authors acknowledge that the system's performance is dependent on the quality and coverage of the training dataset. While the model is designed to be generalizable to any synthesizer, its effectiveness will vary based on the specific synthesis algorithm and the richness of its parameter space. Future work could involve exploring more sophisticated generative models, expanding the range of control modalities, and further investigating the semantic interpretability of the learned latent dimensions.

Summary updated

BibTeX
@inproceedings{esling2020flowsynth,
  author = {Philippe Esling and Naotake Masuda and Axel Chemla-Romeu-Santos},
  title = {{FlowSynth: Simplifying Complex Audio Generation Through Explorable Latent Spaces with Normalizing Flows}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/767},
}

Loading related content...

Figure