Flat Latent Manifolds for Human-machine Co-creation of Music

Nutan Chen, Djalel Benbouzid, Francesco Ferroni, Mathis Nitschke, Luciano Pinna, Patrick van der Smagt

TL;DR

This paper introduces FM-Music-VAE, a novel approach to music generation that creates smoother, more interpretable latent spaces for human-machine co-creation. By regularizing the latent space to be a "flat" manifold, their model enables intuitive musical interpolations and interactive jam sessions with human musicians.

What Problem Does This Paper Address?

Generating compelling and coherent music with AI is challenging. While Variational Auto-Encoders (VAEs) can learn to generate sequences, standard VAEs often produce abrupt, unnatural changes when interpolating between musical ideas in their latent space. This lack of smoothness hinders their use in interactive music creation where a human musician might want to intuitively guide the AI's output. The paper aims to bridge this gap by creating a VAE that generates musically meaningful and smooth transitions, fostering a more natural interplay between human and machine.

Methods

The core of the paper is the development of FM-Music-VAE, a recurrent VAE designed for music generation. The key innovation lies in regularizing the latent space to have a "flat" Riemannian manifold, meaning it behaves locally like a standard Euclidean space. This is achieved by incorporating a metric tensor regularization term into the VAE's loss function. This regularization encourages the decoder to be distance-preserving, ensuring that linear interpolations in the latent space translate to smooth, realistic musical changes. They also employ a hierarchical prior (VHP) to combat over-regularization and improve latent representation quality.

Datasets

The authors evaluated their model on three music datasets:

  • Drum/Groove Dataset: Consisting of two and four-bar sequences with information on MIDI notes, velocity, and timing offsets.
  • Piano Dataset: Derived from the Ryan's Mammoth dataset, augmented with pitch shifting.
  • Cello Dataset: From the Bach Cello Suites, also augmented with pitch shifting.

These datasets were used for training, validation, and testing, with specific splits detailed in the paper.

Key Insights and Ideas

  • Flat Latent Manifolds for Smoothness: The central idea is that a "flat" latent space is crucial for generating smooth musical transitions. This is a departure from standard VAEs where the latent space can be highly curved, leading to jarring changes.
  • Interpretability through Structure: By enforcing a flat manifold, the latent space becomes more structured and interpretable. This allows human musicians to intuitively understand and manipulate the AI's musical output.
  • Human-Centric Design: The work emphasizes a human-centered approach to AI model design, prioritizing interpretability and interactive capabilities for the end-user (the musician).
  • Beyond Reconstruction Loss: The paper argues for evaluating generative models not just on reconstruction accuracy but also on the quality of interaction and interpretability, especially in creative domains like music.

Key Results

  • Improved Smoothness: FM-Music-VAE consistently achieved better smoothness in interpolations compared to MusicVAE across drum and piano datasets, especially with lower-dimensional latent spaces.
  • Enhanced Interpolation Quality: Quantitative evaluations showed FM-Music-VAE outperforming MusicVAE in metrics like density, velocity, and pitch for drum and piano datasets. For instance, on the drum dataset (10% percentile), FM-Music-VAE achieved a density score of 0.039 ± 0.023, compared to MusicVAE's 0.051 ± 0.031.
  • Structured Latent Space: The latent space of FM-Music-VAE demonstrated better structure, with attributes like density showing clearer correlations with latent space directions.
  • Qualitative Success in Live Jam Session: A live interactive jam session with a professional drummer provided qualitative evidence that the latent representation was interpretable and usable by a non-expert, enabling a meaningful musical dialogue.

Limitations and Future Work

  • Limitations: While the live performance was qualitatively successful, the authors acknowledge that a more thorough and scientific study is needed to definitively support the claim of intuitive human usability. The current work focuses on specific musical attributes and might not capture all nuances of musical expression.
  • Future Work: The authors suggest exploring high-level control in the latent space, potentially using reinforcement learning, to generate even longer and more complex musical sequences. Further investigation into the interpretability and usability of the latent space by a wider range of musicians is also a natural next step.

Summary updated

Abstract

The use of machine learning in artistic music generation leads to controversial discussions of the quality of art, for which objective quantification is nonsensical. We therefore consider a music-generating algorithm as a counterpart to a human musician, in a setting where reciprocal interplay is to lead to new experiences, both for the musician and the audience. To obtain this behaviour, we resort to the framework of recurrent Variational Auto-Encoders (VAE) and learn to generate music, seeded by a human musician. In the learned model, we generate novel musical sequences by interpolation in latent space. Standard VAEs however do not guarantee any form of smoothness in their latent representation. This translates into abrupt changes in the generated music sequences. To overcome these limitations, we regularise the decoder and endow the latent space with a flat Riemannian manifold, i.e., a manifold that is isometric to the Euclidean space. As a result, linearly interpolating in the latent space yields realistic and smooth musical changes that fit the type of machine–musician interactions we aim for. We provide empirical evidence for our method via a set of experiments on music datasets and we deploy our model for an interactive jam session with a professional drummer. The live performance provides qualitative evidence that the latent representation can be intuitively interpreted and exploited by the drummer to drive the interplay. Beyond the musical application, our approach showcases an instance of human-centred design of machine-learning models, driven by interpretability and the interaction with the end user.
BibTeX
@inproceedings{chen2022flat,
  author = {Nutan Chen and Djalel Benbouzid and Francesco Ferroni and Mathis Nitschke and Luciano Pinna and Patrick van der Smagt},
  title = {{Flat Latent Manifolds for Human-machine Co-creation of Music}},
  booktitle = {AIMC},
  year = {2022},
}

Loading related content...

Figure