Flat latent manifolds for music improvisation between human and machine

Nutan Chen, Djalel Benbouzid, Francesco Ferroni, Mathis Nitschke, Luciano Pinna, Patrick van der Smagt

Abstract

The use of machine learning in artistic music generation leads to controversial discussions of the quality of art, for which objective quantification is nonsensical. We therefore consider a music-generating algorithm as a counterpart to a human musician, in a setting where reciprocal interplay is to lead to new experiences, both for the musician and the audience. To obtain this behaviour, we resort to the framework of recurrent Variational Auto-Encoders (VAE) and learn to generate music, seeded by a human musician. In the learned model, we generate novel musical sequences by interpolation in latent space. Standard VAEs however do not guarantee any form of smoothness in their latent representation. This translates into abrupt changes in the generated music sequences. To overcome these limitations, we regularise the decoder and endow the latent space with a flat Riemannian manifold, i.e., a manifold that is isometric to the Euclidean space. As a result, linearly interpolating in the latent space yields realistic and smooth musical changes that fit the type of machine--musician interactions we aim for. We provide empirical evidence for our method via a set of experiments on music datasets and we deploy our model for an interactive jam session with a professional drummer. The live performance provides qualitative evidence that the latent representation can be intuitively interpreted and exploited by the drummer to drive the interplay. Beyond the musical application, our approach showcases an instance of human-centred design of machine-learning models, driven by interpretability and the interaction with the end user.

Introduction

The intersection of machine learning and artistic creation often sparks debate regarding the quantification of "artistic quality." Rather than attempting to automate art, this work positions the machine as a collaborative partner for human musicians. The goal is to facilitate a reciprocal interplay that generates novel musical experiences. To achieve this, we utilize recurrent Variational Auto-Encoders (VAEs) to generate musical sequences seeded by human input. However, a significant hurdle in standard VAEs is the lack of smoothness in their latent representations, leading to disjointed and unmusical transitions during interpolation. We address this by regularizing the decoder to ensure the latent space behaves as a flat Riemannian manifold, enabling intuitive and predictable human-machine interaction.

Methodology

The core of our approach is the FM-Music-VAE, which builds upon the hierarchical MusicVAE architecture. The model employs Gated Recurrent Units (GRUs) in both the encoder and decoder. To manage long-term structure, a "conductor" GRU produces embeddings for lower-level recurrent networks that generate the actual note sequences.

The hierarchical architecture of the MusicVAE model, featuring an encoder and a conductor-based decoder.

The FM-Music-VAE architecture is designed to handle the multi-level nature of music. The encoder processes input sequences (e.g., drum patterns) using bidirectional GRUs to create a latent representation $z$. The decoder is hierarchical: a high-level "conductor" GRU breaks down the latent vector into several segment embeddings. These embeddings then seed lower-level GRUs that generate the final sequence of notes or drum hits. This hierarchy allows the model to maintain coherence over multiple bars while capturing fine-grained details.

To enforce a "flat" geometry, we introduce three primary regularizers:

  1. Flat Riemannian Manifolds: We constrain the latent space to be isometric to Euclidean space. This ensures that linear paths in the latent space correspond to smooth, minimal changes in the musical output.
  2. Variational Hierarchical Priors (VHP): Instead of a standard Gaussian prior, we use a two-layer stochastic prior to better capture complex musical distributions and prevent posterior collapse.
  3. Latent Space Mixup: By interpolating between data points during training, we ensure the manifold remains smooth even in regions sparsely populated by training data.

Experimental Results

We evaluated our model on the Groove dataset (MIDI drums), Ryan’s Mammoth Collection (piano melodies), and Bach’s Cello Suites. A key metric was "smoothness," measured by the Hamming distance between consecutive generated sequences during latent space interpolation.

Comparison of smoothness metrics across different model configurations and latent space dimensions. Lower values indicate smoother transitions.

As shown in Figure 2, FM-Music-VAE consistently achieves lower smoothness scores (indicating higher musical continuity) compared to the baseline MusicVAE. Notably, our model in 32 dimensions outperformed the standard model even when the latter used 512 dimensions, demonstrating the efficiency of a well-structured latent space. The quantitative evaluation also focused on predictability. We measured how musical attributes like velocity and note density changed along specific vectors in the latent space. FM-Music-VAE showed a Pearson correlation of up to 0.91 for note density in the Cello dataset, significantly higher than the baseline. This means a musician can reliably "turn up" the intensity of the AI's playing by moving in a known direction.

Live Interaction and Conclusion

The practical utility of FM-Music-VAE was demonstrated in a live jam session with a professional drummer. The drummer could influence the AI's responses by navigating the latent space, reporting a tangible sense of musical trajectory. This work highlights the importance of human-centered design in machine learning, where mathematical properties like manifold flatness directly translate into user interpretability and creative agency. Future work will explore longer sequence lengths and reinforcement learning for strategic musical interaction.

Summary updated

Abstract

The use of machine learning in artistic music generation leads to controversial discussions of the quality of art, for which objective quantification is nonsensical. We therefore consider a music-generating algorithm as a counterpart to a human musician, in a setting where reciprocal interplay is to lead to new experiences, both for the musician and the audience. To obtain this behaviour, we resort to the framework of recurrent Variational Auto-Encoders (VAE) and learn to generate music, seeded by a human musician. In the learned model, we generate novel musical sequences by interpolation in latent space. Standard VAEs however do not guarantee any form of smoothness in their latent representation. This translates into abrupt changes in the generated music sequences. To overcome these limitations, we regularise the decoder and endow the latent space with a flat Riemannian manifold, i.e., a manifold that is isometric to the Euclidean space. As a result, linearly interpolating in the latent space yields realistic and smooth musical changes that fit the type of machine--musician interactions we aim for. We provide empirical evidence for our method via a set of experiments on music datasets and we deploy our model for an interactive jam session with a professional drummer. The live performance provides qualitative evidence that the latent representation can be intuitively interpreted and exploited by the drummer to drive the interplay. Beyond the musical application, our approach showcases an instance of human-centred design of machine-learning models, driven by interpretability and the interaction with the end user.
BibTeX
@inproceedings{chen2022flat,
  author = {Nutan Chen and Djalel Benbouzid and Francesco Ferroni and Mathis Nitschke and Luciano Pinna and Patrick van der Smagt},
  title = {{Flat latent manifolds for music improvisation between human and machine}},
  booktitle = {AIMC},
  year = {2022},
}
Figure