A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music

Adam Roberts, Jesse Engel, Colin Raffel, Curtis Hawthorne, Douglas Eck

Deep latent variable models, particularly Variational Autoencoders (VAEs), have proven highly effective for modeling images. They enable creative applications such as latent space interpolation (morphing one image into another) and attribute arithmetic (e.g., adding "sunglasses" to a face). However, applying these models to long sequences of discrete tokens—like musical notes or text—has historically been notoriously difficult.

Standard recurrent VAEs typically employ a "flat" RNN decoder that generates tokens autoregressively, predicting the next token based on previous ones. Because the RNN is so powerful on its own, it often learns to ignore the latent code entirely, a phenomenon known as "posterior collapse." Consequently, the model forgets the global context over long sequences. This makes it impossible to generate coherent long-term structures, such as a 16-bar melody, or to perform meaningful latent space interpolations, as the model simply relies on local, step-by-step predictions.

To address this, researchers introduce MusicVAE, a recurrent VAE designed specifically to model long sequences of discrete data by employing a novel hierarchical decoder.

The MusicVAE Architecture

MusicVAE alters the standard sequence-to-sequence VAE architecture to force the model to utilize its latent space, preventing posterior collapse.

MusicVAE Architecture
  • Bidirectional Encoder: The model uses a standard 2-layer bidirectional LSTM to process an entire input sequence into a single 512-dimensional latent vector $z$.
  • Hierarchical Decoder: Instead of a single RNN generating the whole sequence, the latent vector $z$ is fed into a "conductor" RNN. This conductor outputs a sequence of embedding vectors, one for each subsequence (e.g., one per musical bar). Each embedding then initializes a shared bottom-level decoder RNN that autoregressively generates the actual notes for just that specific bar.
  • Multi-Stream Modeling: The authors also introduce a "trio" model that extends this hierarchy orthogonally. The conductor passes embeddings to three separate bottom-level decoders to simultaneously generate a melody, a bassline, and a drum pattern.

By restricting the bottom-level decoder's scope to a single bar, the model cannot rely purely on autoregression to coast through the whole song. It is forced to rely on the conductor's embeddings—and by extension, the latent code—to maintain long-term coherence.

Key Insights

  • Constraining the Decoder: Artificially limiting the temporal scope of the bottom-level RNN elegantly solves the posterior collapse problem without needing complex loss-weighting schemes.
  • Domain-Aligned Architecture: The hierarchical choice aligns perfectly with the natural structure of music. Songs are built of sections, which are built of bars, which are built of beats. The conductor/decoder split maps directly to this bar/beat structure.
  • Smooth Latent Spaces for Discrete Data: Despite music being a sequence of discrete, categorical tokens, MusicVAE learns a remarkably smooth continuous latent space. Interpolating between two distinct 16-bar melodies yields intermediate sequences that are harmonically and rhythmically coherent, rather than a dissonant mashup of notes.
  • Latent Attribute Arithmetic: Just like visual VAEs, users can compute an "add syncopation" or "increase note density" vector in MusicVAE's latent space and apply it to existing melodies with predictable, musical results.

Experimental Results

The models were trained on a proprietary dataset of ~1.5 million unique web-scraped MIDI files, extracting 2-bar and 16-bar monophonic melodies, drum patterns, and 16-bar "trio" sequences.

Reconstruction Accuracy: The hierarchical decoder drastically reduces "exposure bias" (the drop in accuracy when switching from teacher-forced training to autoregressive sampling). For 16-bar melodies, the flat baseline model's accuracy dropped from 88.3% (teacher-forced) to just 62.0% (sampled). The hierarchical model maintained high accuracy, dropping only from 91.9% to 81.2%. For 16-bar drum tracks in the trio setting, the hierarchical model achieved 86.3% sampling accuracy compared to the flat model's 64.1%.

Human Evaluation: In blind A/B tests (192 ratings per study), listeners overwhelmingly preferred samples from the hierarchical model over the flat model across melodies, trios, and drums.

Human Listening Test Results

Remarkably, participants actually preferred the hierarchical model's generated drum patterns slightly more often than real human-composed drum data (though the difference wasn't statistically significant). The authors suspect this is because the generated samples had more engaging variety than the highly repetitive real drum loops.

Limitations and Future Work

While highly effective, the model has limitations regarding entangled attributes. The heuristic-based attribute vectors sometimes capture overlapping characteristics; for example, artificially increasing "note density" in the latent space inadvertently decreases "8th note syncopation."

Future work will explore evaluating attribute vector manipulations for more complex, non-trivial musical labels (like ornamentation or call/response structures) that are harder to compute heuristically. Furthermore, the authors express strong interest in applying this hierarchical recurrent VAE architecture to other types of long sequential data beyond music, such as text or code.

Summary updated

Abstract

The Variational Autoencoder (VAE) has proven to be an effective model for producing semantically meaningful latent representations for natural data. However, it has thus far seen limited applicatio...
BibTeX
@inproceedings{roberts2018hierarchical,
  author = {Adam Roberts and Jesse Engel and Colin Raffel and Curtis Hawthorne and Douglas Eck},
  title = {{A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music}},
  booktitle = {International Conference on Machine Learning},
  year = {2018},
  url = {https://openreview.net/forum?id=HkZlSoWdZH},
}
Figure