Rhythm, Chord and Melody Generation for Lead Sheets using Recurrent Neural Networks

Cedric De Boom, Stephanie Van Laere, Tim Verbelen, Bart Dhoedt

Abstract

Generating musically coherent lead sheets—comprising chords, rhythm, and melody—remains a significant challenge for artificial intelligence. This paper introduces a novel two-stage generation approach using Recurrent Neural Networks (RNNs). The first stage generates a structural "template" of chords and rhythms, while the second stage employs Bidirectional Long Short-Term Memory (BiLSTM) networks to populate this template with melody notes. By allowing the melody generator to "look ahead" at the entire harmonic sequence, the model produces music with enhanced logical flow and coherence. Subjective listening tests demonstrate that the proposed model significantly outperforms one-stage baselines and approaches human-level performance when conditioned on existing chord schemes.

Introduction

The creation of music that resonates with human listeners requires a deep understanding of the interdependence between harmony, melody, and rhythm. Traditional end-to-end models often struggle with musical coherence, producing aimless melodies that fail to align with underlying chord changes. This research addresses the problem by mimicking the human songwriting process: establishing a rhythmic and harmonic "groove" before composing a melody on top.

Methodology

The core of the proposed system is a two-stage generation process. In Stage One, an LSTM-based model generates a sequence of rhythm patterns and chord symbols, creating the "skeleton" of the lead sheet. In Stage Two, a melody is generated based on this template.

The two-stage RNN architecture. Stage one (left) generates the rhythm/chord template, while stage two (right) uses BiLSTMs to generate the melody conditioned on that template.

The use of Bidirectional LSTMs (BiLSTMs) in the second stage is a key innovation. Unlike standard LSTMs, BiLSTMs process the harmonic sequence in both forward and backward directions, enabling the melody generator to anticipate future chord changes. This "foresight" is crucial for creating melodies that feel integrated with the harmony.

Data Representation

To facilitate machine learning, musical elements are encoded into discrete vectors. The model uses one-hot encoding for 12 rhythm types, 48 chord options (12 roots across 4 modes), and 130 melody states (MIDI pitches, rests, and barlines).

Example of the data representation mapping musical notation to discrete chord ($c_i$), rhythm ($r_i$), and melody ($m_i$) tokens.

Datasets and Preprocessing

The models were trained on the Wikifonia Dataset, containing over 6,500 lead sheets. Preprocessing involved simplifying chord modes to four essentials (major, minor, diminished, augmented) and ensuring monophony by retaining only the highest pitch. Data augmentation via transposition into all 12 keys was used to expand the dataset and prevent key bias.

Results and Discussion

Subjective evaluation involved 40 participants rating generated music against human compositions. The two-stage BiLSTM model significantly outperformed one-stage baselines in both "Pleasingness" and "Coherence." Notably, when the model was provided with existing chord schemes from popular songs, its "Turing" score—a measure of how human the music sounds—was nearly indistinguishable from actual human-composed pieces. Interestingly, the researchers observed higher disagreement among listeners for "good" music than for "bad" music, suggesting that while technical flaws are universally recognized, musical beauty remains subjective.

Conclusion

This research highlights the importance of hierarchical structure in music generation. By separating global structure from local detail and utilizing bidirectional context, the model achieves a high degree of musicality. Future work will explore joint training of the two stages and expanding the model's vocabulary to support more complex genres and polyphonic textures.

Summary updated

BibTeX
@inproceedings{boom2019rhythm,
  author = {Cedric De Boom and Stephanie Van Laere and Tim Verbelen and Bart Dhoedt},
  title = {{Rhythm, Chord and Melody Generation for Lead Sheets using Recurrent Neural Networks}},
  booktitle = {PKDD/ECML Workshops},
  year = {2019},
  url = {https://openreview.net/forum?id=HskdIhpwQXa},
}
Figure