Abstract
With recent breakthroughs in artificial neural networks, deep generative models have become one of the leading techniques for computational creativity. Despite very promising progress on image and short sequence generation, symbolic music generation remains a challenging problem since the structure of compositions are usually complicated. In this study, we attempt to solve the melody generation problem constrained by the given chord progression. In particular, we explore the effect of explicit architectural encoding of musical structure via comparing two sequential generative models: LSTM (a type of RNN) and WaveNet (dilated temporal-CNN). As far as we know, this is the first study of applying WaveNet to symbolic music generation, as well as the first systematic comparison between temporal-CNN and RNN for music generation. We conduct a survey for evaluation in our generations and implemented Variable Markov Oracle in music pattern discovery. Experimental results show that to encode structure more explicitly using a stack of dilated convolution layers improved the performance significantly, and a global encoding of underlying chord progression into the generation procedure gains even more.
Introduction
Generating musically coherent melodies is a significant challenge for artificial intelligence. Unlike simple sequences, music is defined by hierarchical structures, long-term dependencies, and motifs that must align with a global harmonic framework. While models have succeeded in mimicking local textures, long-term coherence often remains elusive. This research investigates whether musical structure is better learned implicitly through internal memory (LSTMs) or if it should be "baked into" the network architecture itself (WaveNet).
Methodology
The researchers framed the task as conditioned melody generation, where a melody is generated to fit a pre-defined chord progression. Three primary architectures were evaluated:
- Unidirectional LSTM: A standard recurrent model that processes notes sequentially, with access only to current and past chord information.
- Bidirectional LSTM (Bi-LSTM): An advanced recurrent architecture that processes the chord progression in both directions, allowing the model to "look ahead" at future harmonic goals.
- WaveNet (Dilated Temporal-CNN): Adapted from raw audio generation to symbolic MIDI-like data. WaveNet utilizes dilated convolutions to provide a massive receptive field, enabling the model to observe a wide window of history without excessive computational cost.
Data Representation
To handle the complexities of pitch and rhythm, the authors employed a "staggered representation." This encoding accounts for 130 dimensions for melody (including pitch, rest states, and holding states to distinguish sustained notes) and 25 dimensions for chord information.
Experimental Setup
The models were trained on the Nottingham Database, consisting of 941 folk songs. Data was augmented via transposition into all 12 keys and normalized to a fixed 120 BPM with a 1/16th note frame size. Evaluation was conducted through two lenses: a subjective human survey of 106 participants and an objective pattern discovery analysis using the Variable Markov Oracle (VMO) algorithm.
Key Insights and Results
The core finding of the study is the superiority of explicit structural encoding. WaveNet’s architecture naturally mirrors musical hierarchy: its layers of dilated convolutions capture relationships across expanding time scales (e.g., 1/16 notes to 1/8 notes). This allows for superior rhythmic mastery, including the generation of complex patterns like triplets.
However, the Bi-LSTM emerged as the overall winner in human preference. The researchers attribute this to the "teleological" nature of music—it moves toward a goal. By encoding the entire chord progression as a global constraint, the Bi-LSTM generates notes that resolve satisfyingly into upcoming chord changes. Objective VMO analysis confirmed these results, showing that both Bi-LSTM and WaveNet produced significantly more stable motifs than the standard Unidirectional LSTM. Statistical significance was robust, with ANOVA tests yielding p-values as low as $3.02e^{-16}$.
Conclusion
This study demonstrates that the way we encode musical time and global constraints significantly impacts the coherence of AI-generated music. While WaveNet excels at hierarchical rhythmic structures, the Bi-LSTM's ability to "see the future" of a chord progression makes it uniquely suited for harmonic alignment. Future work may explore "Bidirectional WaveNet" architectures to combine the strengths of both approaches.
Summary updated