Abstract
The Variational Autoencoder (VAE) has proven to be an effective model for producing semantically meaningful latent representations for natural data. However, it has thus far seen limited application to sequential data, and, as we demonstrate, existing recurrent VAE models have difficulty modeling sequences with long-term structure. To address this issue, we propose the use of a hierarchical decoder, which first outputs embeddings for subsequences of the input and then uses these embeddings to generate each subsequence independently. This structure encourages the model to utilize its latent code, thereby avoiding the "posterior collapse" problem, which remains an issue for recurrent VAEs. We apply this architecture to modeling sequences of musical notes and find that it exhibits dramatically better sampling, interpolation, and reconstruction performance than a "flat" baseline model.
The Challenge of Long-Term Structure
Generative models like VAEs are highly effective at creating realistic images because they compress data into a "latent space"—a mathematical map where similar objects are grouped together. However, applying this to music is notoriously difficult.
The primary challenge is posterior collapse. In a standard VAE, the decoder is often a powerful Recurrent Neural Network (RNN). If the decoder is too proficient at predicting the next note based only on local context (the previous few notes), it learns to ignore the latent code—the "blueprint" provided by the encoder—entirely. Furthermore, in long sequences like a 16-bar melody, the influence of the initial latent state tends to fade over time due to vanishing gradients. The result is a model that produces locally coherent notes but lacks any global sense of melody, repetition, or structure.
The MusicVAE Architecture
To bridge the gap between local detail and global structure, we propose MusicVAE, featuring a novel hierarchical decoder. This architecture decomposes the generation process into high-level planning and low-level execution.
- Bidirectional Encoder: A two-layer bidirectional LSTM processes the entire musical sequence and compresses it into a single latent vector $z$.
- The Conductor RNN: Instead of generating notes directly, the latent vector $z$ initializes a "Conductor" RNN. This high-level network produces a sequence of embeddings—one for each "subsequence" (e.g., one embedding for each bar of music).
- Sub-layer Decoders: Each embedding from the Conductor initializes a separate, lower-level RNN. These sub-decoders are responsible for generating the actual notes for their specific bar. Crucially, these sub-decoders have a limited scope; they cannot "see" across bar boundaries, forcing them to rely on the Conductor's embeddings for long-term context.
By limiting the power of the bottom-level decoders and preventing them from communicating with each other, the model is forced to use the latent code to maintain structure. This effectively solves the posterior collapse problem for long sequences.
Experimental Results
We trained MusicVAE on massive collections of MIDI data, including 2-bar and 16-bar sequences of monophonic melodies, drum patterns, and three-instrument "trios."
Reconstruction and Human Preference
On 16-bar melodies, the hierarchical MusicVAE achieved a reconstruction accuracy of 81.2%, compared to just 62.0% for a standard "flat" VAE baseline. This quantitative improvement was mirrored in qualitative blind listening tests.
As shown in the listening tests, human participants overwhelmingly preferred samples from the hierarchical model over the flat baseline. Remarkably, for drum patterns, listeners rated MusicVAE samples as "more musical" nearly as often as they did for real, human-composed data.
Latent Space Arithmetic and Interpolation
Because the model successfully utilizes its latent code, we can perform "latent space arithmetic." For example, one can take a "no-syncopation" vector and add it to a melody to increase its rhythmic complexity. Furthermore, MusicVAE enables smooth semantic interpolation. Unlike "data-space" blending, which creates a dissonant mess of overlapping notes, MusicVAE's latent interpolation creates intermediate melodies that share the stylistic DNA of both endpoints while remaining musically coherent throughout the transition.
Conclusion and Future Work
MusicVAE demonstrates that hierarchical structure is essential for capturing long-term dependencies in sequential data. By enforcing independence between sub-decoders, we compel the model to utilize its global latent representation. While modeling complex "call and response" interactions between multiple players remains a challenge, this hierarchical approach is generalizable and could be applied to other long-form sequential data, such as multi-paragraph text generation or long-term robotic planning.
Summary updated