Continuous Melody Generation via Disentangled Short-Term Representations and Structural Conditions

Ke Chen, Gus Xia, Shlomo Dubnov

TL;DR

This paper introduces a two-stage generative framework that creates long-term melodies by disentangling pitch and rhythm. By combining a Variational Autoencoder (EC2-VAE) with a Bi-LSTM, the system can "continue" a musical context while adhering to specific rhythmic templates and harmonic constraints, resulting in AI-generated music with human-like structural consistency.

What Problem Does This Paper Address?

While deep learning has made massive strides in music generation, most models still struggle with long-term structure. A model might generate a sequence of notes that sounds pleasant in the short term but lacks the "connective tissue"—the repetitions, rhythmic motives, and harmonic arcs—that define human compositions.

Existing approaches often fall into two traps: they are either too computationally expensive (like some Transformers) or too rigid (like rule-based systems). Furthermore, many models suffer from "rhythmic blindness," where the generated rhythm becomes chaotic or loses its "groove" over time. The authors aim to bridge this gap by allowing users to specify structural conditions (like a specific rhythm pattern or a sequence of chord functions) to guide the generation process.

Methods

The authors propose a two-stage architecture:

  1. Representation Stage (EC2-VAE): They utilize the EC2-VAE (Explicitly Conditioned VAE) to compress 8-beat note sequences into a 256-dimensional latent vector ($z$). Crucially, this vector is disentangled into two independent components: $z_p$ (representing pitch contour) and $z_r$ (representing rhythm patterns). This allows the model to manipulate the "what" (notes) and the "when" (rhythm) separately.
  2. Generative Stage (Bi-LSTM): Instead of predicting music note-by-note, an 8-layer Bidirectional LSTM predicts the latent vectors of future 8-beat segments. By working at the level of "musical sentences" rather than individual "words," the model maintains better coherence.
  3. Structural Conditioning: The generation is guided by two global conditions:
  4. Chord Functions: A simplified 4-label system (Tonic, Dominant, Subdominant, and Other) that represents harmonic tension and resolution.
  5. Rhythm Templates: The model can "imitate" the rhythm of an existing song by using its $z_r$ vectors as a condition for generating new pitch contours.

Datasets

The model was trained on the Nottingham Database, a collection of 1,200 British and American folk songs.

  • Preprocessing: The data was augmented by transposing songs into all 12 tonalities and normalized to a standard 120 BPM.
  • Representation: Melodies were converted into 130-dimensional one-hot vectors (128 pitches + "rest" + "hold"), and chords were represented as 12-dimensional chromatic features.

Key Insights and Ideas

  • Disentanglement as a Control Knob: The most powerful idea here is that by separating rhythm from pitch in the latent space, you can perform "rhythm transfer." You can take the "groove" of Song A and the "harmonic skeleton" of Song B to generate a brand-new Melody C that feels structurally related to both.
  • Macro-level Prediction: By predicting latent vectors of 8-beat phrases rather than individual notes, the model avoids the "drift" common in autoregressive models. It thinks in musical phrases, which is much closer to how human composers work.
  • Semantic Anchoring: Using high-level "chord functions" (T, D, S, O) provides a low-dimensional way to guide the emotional arc of the music without over-constraining the specific notes the AI chooses to play.

Key Results

The authors evaluated the model against two LSTM-based baselines using both quantitative and qualitative metrics:

  • Rhythm Accuracy: Using a novel duration-ratio-based algorithm, the proposed model achieved a 0.981 accuracy score, significantly higher than the baselines (~0.77-0.80), proving it successfully "memorized" and applied rhythmic patterns.
  • Subjective Listening Study: 50 participants rated the generations. The proposed model scored highest in Structure (3.62/5) and Overall Musicality (3.46/5).
  • Visual Analysis (VMO): Using Variable Markov Oracle (VMO) analysis, the authors showed that their model creates melodies with longer "blocks" of repeating patterns and motives, mimicking the structural "echoes" found in original folk songs.

Limitations and Future Work

  • Manual Labeling: The current system relies on manually labeled chord functions. The authors suggest that future iterations should move toward fully automated semantic understanding of musical content.
  • Monophonic Focus: The study focuses primarily on monophonic melodies with chordal accompaniment. Extending this disentanglement approach to complex polyphonic textures (multiple independent voices) remains a challenge.
  • Interactive Design: While the paper mentions interactive performance (e.g., tapping a rhythm for the piano to follow), the current implementation is more of a "batch" generation tool. Real-time interactive applications are a suggested next step.

Summary updated

Abstract

Automatic music generation is an interdisciplinary research topic that combines computational creativity and semantic analysis of music to create automatic machine improvisations. An important property of such a system is allowing the user to specify conditions and desired properties of the generated music. In this paper we designed a model for composing melodies given a user specified symbolic scenario combined with a previous music context. We add manual labeled vectors denoting external music quality in terms of chord function that provides a low dimensional representation of the harmonic tension and resolution. Our model is capable of generating long melodies by regarding 8-beat note sequences as basic units, and shares consistent rhythm pattern structure with another specific song. The model contains two stages and requires separate training where the first stage adopts a Conditional Variational Autoencoder (C-VAE) to build a bijection between note sequences and their latent representations, and the second stage adopts long short-term memory networks (LSTM) with structural conditions to continue writing future melodies. We further exploit the disentanglement technique via C-VAE to allow melody generation based on pitch contour information separately from conditioning on rhythm patterns. Finally, we evaluate the proposed model using quantitative analysis of rhythm and the subjective listening study. Results show that the music generated by our model tends to have salient repetition structures, rich motives, and stable rhythm patterns. The ability to generate longer and more structural phrases from disentangled representations combined with semantic scenario specification conditions shows a broad application of our model.
BibTeX
@inproceedings{chen2020continuous,
  author = {Ke Chen and Gus Xia and Shlomo Dubnov},
  title = {{Continuous Melody Generation via Disentangled Short-Term Representations and Structural Conditions}},
  booktitle = {ICSC},
  year = {2020},
  url = {https://openreview.net/forum?id=xe4UcFbu0TQ},
}

Loading related content...

Figure