Inspecting and Interacting with Meaningful Music Representations using VAE

Ruihan Yang, Tianyao Chen, Yiyi Zhang, Gus Xia

Abstract

Variational Autoencoders (VAEs) have already achieved great results on image generation and recently made promising progress on music generation. However, the generation process is still quite difficult to control in the sense that the learned latent representations lack meaningful music semantics. It would be much more useful if people can modify certain music features, such as rhythm and pitch contour, via latent representations to test different composition ideas. In this paper, we propose a new method to inspect the pitch and rhythm interpretations of the latent representations and we name it disentanglement by augmentation. Based on the interpretable representations, an intuitive graphical user interface is designed for users to better direct the music creation process by manipulating the pitch contours and rhythmic complexity.

Introduction

Generative models like MusicVAE have shown incredible promise in creating symbolic music (MIDI). However, these models typically map music into a high-dimensional latent space that lacks human-readable meaning. If a composer wants to make a melody "more rhythmic" or "higher in pitch," they don't know which specific numbers in that vector to change. The challenge is disentanglement: separating the tangled web of musical features so that changing one (like rhythm) doesn't accidentally warp another (like melody). The authors aim to turn the "black box" of VAEs into a "radio" with labeled knobs that composers can actually use.

Methods and Architecture

The researchers built their work on the MusicVAE architecture, which uses Bidirectional Gated Recurrent Units (GRUs) to encode and decode 2-bar MIDI clips. As illustrated in the diagram below, the model maps inputs to a latent representation $z$.

The MusicVAE architecture used for encoding and decoding 2-bar MIDI sequences.

Their approach involves three key steps:

  1. Dimensionality Reduction: They compressed the standard 512-dimension latent space down to 128 dimensions using PCA, capturing over 99% of the data's variance.
  2. $\beta$-VAE Training: They used a $\beta$-VAE framework with a low $\beta$ value (0.1) to prioritize accurate reconstruction over the randomness typically encouraged in standard VAEs.
  3. Disentanglement by Augmentation: This is the paper's core novelty. To find which dimensions control which features, they took a music clip, applied a known transformation (like transposing it up a few semitones), and then looked at which dimensions in the latent vector changed the most. By averaging these changes across thousands of samples, they identified the "significant dimensions" for specific musical traits.

Key Insights: The Sparsity of Semantics

One of the most striking findings is how few dimensions actually matter for core musical features. Out of 128 dimensions, the researchers found that only two dimensions were responsible for pitch contour and only five were responsible for rhythmic complexity.

Identification of latent dimensions sensitive to pitch transformations, showing a sharp drop-off in sensitivity after the first few indices.

Perhaps most importantly, the dimensions for pitch and rhythm did not overlap. This suggests that the VAE naturally learns to separate these concepts even without being explicitly told to do so. Furthermore, the latent representations changed approximately linearly as the pitch or rhythm was modified, allowing for predictable control via simple sliders.

Key Results

By increasing the values of the two identified "pitch dimensions," the researchers successfully generated new versions of Mozart’s Twinkle, Twinkle, Little Star that had higher pitch registrations while maintaining the original rhythm. Similarly, adjusting the five "rhythm dimensions" allowed them to increase note density while preserving the general melodic contour. The authors also demonstrated a grid-based interpolation (using SLERP) to create a matrix where the X-axis transitioned the rhythm and the Y-axis transitioned the pitch, allowing for a hybrid "style-transfer" between two different pieces of music.

Limitations and Future Work

The current method for finding these dimensions is "post-hoc" and manual, requiring prior knowledge of the feature to be augmented. Future work aims to develop more automated "self-disentangling" models. Additionally, while pitch and rhythm are easy to augment, more complex musical concepts like "mood" or "tension" are harder to define mathematically. Finally, while the model excels at 2-bar clips, handling longer, more complex musical structures remains a challenge for the hierarchical VAE architecture.

Summary updated

Abstract

Variational Autoencoders(VAEs) have already achieved great results on image generation and recently made promising progress on music generation. However, the generation process is still quite difficult to control in the sense that the learned latent representations lack meaningful music semantics. It would be much more useful if people can modify certain music features, such as rhythm and pitch contour, via latent representations to test different composition ideas. In this paper, we propose a new method to inspect the pitch and rhythm interpretations of the latent representations and we name it disentanglement by augmentation. Based on the interpretable representations, an intuitive graphical user interface is designed for users to better direct the music creation process by manipulating the pitch contours and rhythmic complexity.
BibTeX
@inproceedings{yang2019inspecting,
  author = {Ruihan Yang and Tianyao Chen and Yiyi Zhang and Gus Xia},
  title = {{Inspecting and Interacting with Meaningful Music Representations using VAE}},
  booktitle = {NIME},
  year = {2019},
  url = {https://openreview.net/forum?id=5537qix2tF},
}
Figure