Abstract
Automatic music generation is an interdisciplinary research topic that combines computational creativity and semantic analysis of music to create automatic machine improvisations. An important property of such a system is allowing the user to specify conditions and desired properties of the generated music. In this paper we designed a model for composing melodies given a user specified symbolic scenario combined with a previous music context. We add manual labeled vectors denoting external music quality in terms of chord function that provides a low dimensional representation of the harmonic tension and resolution. Our model is capable of generating long melodies by regarding 8-beat note sequences as basic units, and shares consistent rhythm pattern structure with another specific song. The model contains two stages and requires separate training where the first stage adopts a Conditional Variational Autoencoder (C-VAE) to build a bijection between note sequences and their latent representations, and the second stage adopts long short-term memory networks (LSTM) with structural conditions to continue writing future melodies. We further exploit the disentanglement technique via C-VAE to allow melody generation based on pitch contour information separately from conditioning on rhythm patterns. Finally, we evaluate the proposed model using quantitative analysis of rhythm and the subjective listening study. Results show that the music generated by our model tends to have salient repetition structures, rich motives, and stable rhythm patterns. The ability to generate longer and more structural phrases from disentangled representations combined with semantic scenario specification conditions shows a broad application of our model.
Introduction
While deep learning has made significant strides in music generation, creating long-term structural coherence remains a primary challenge. Most models generate sequences that sound pleasant in isolation but lack the repetitions, rhythmic motives, and harmonic arcs characteristic of human composition. Existing approaches often struggle with "rhythmic blindness" or become computationally prohibitive when modeling long sequences. This research addresses these issues by introducing a framework that disentangles musical components and allows for explicit structural conditioning.
Methodology
The proposed system utilizes a two-stage architecture designed to bridge the gap between short-term note generation and long-term musical structure.
1. Representation Stage: EC2-VAE
The first stage employs an Explicitly Conditioned Variational Autoencoder (EC2-VAE) to compress 8-beat note sequences into a 256-dimensional latent vector ($z$). A key innovation is the disentanglement of this vector into two independent components: $z_p$ (representing pitch contour) and $z_r$ (representing rhythm patterns). This separation allows the model to independently manipulate the melodic shape and the rhythmic "groove."
2. Generative Stage: Bi-LSTM
Rather than predicting music note-by-note, which often leads to "drift," the second stage uses an 8-layer Bidirectional LSTM to predict the latent vectors of future 8-beat segments. By operating at the level of musical phrases rather than individual notes, the model maintains superior long-term coherence and structural consistency.
3. Structural Conditioning
The generation process is guided by two primary global conditions:
- Rhythm Templates: The model can "imitate" the rhythm of an existing song by using its $z_r$ vectors as a condition for generating new pitch contours.
- Chord Functions: To provide a low-dimensional representation of harmonic tension and resolution, the authors use a simplified 4-label system: Tonic (T), Dominant (D), Subdominant (S), and Other (O).
Datasets and Evaluation
The model was trained on the Nottingham Database, consisting of 1,200 folk songs. Melodies were represented as 130-dimensional vectors (including pitch, rest, and hold states), and chords were encoded as 12-dimensional chromatic features.
Quantitative evaluation focused on Rhythm Accuracy, where the proposed model achieved a score of 0.981, significantly outperforming LSTM baselines (~0.77-0.80). In a subjective listening study with 50 participants, the model received high marks for Structure (3.62/5) and Overall Musicality (3.46/5). Visual analysis using Variable Markov Oracles (VMO) confirmed that the generated melodies exhibit the repeating patterns and motives typical of human-composed folk music.
Conclusion
By disentangling pitch and rhythm and predicting at the phrase level, this framework enables the generation of long-term, structurally consistent melodies. The use of semantic anchors like chord functions provides a powerful tool for user-guided composition, paving the way for more interactive and controllable AI music systems.
Summary updated