JamBot: Music Theory Aware Chord Based Generation of Polyphonic Music with LSTMs

Gino Brunner, Yuyi Wang, Roger Wattenhofer, Jonas Wiesendanger

Abstract: We propose a novel approach for the generation of polyphonic music based on LSTMs. We generate music in two steps. First, a chord LSTM predicts a chord progression based on a chord embedding. A second LSTM then generates polyphonic music from the predicted chord progression. The generated music sounds pleasing and harmonic, with only few dissonant notes. It has clear long-term structure that is similar to what a musician would play during a jam session. We show that our approach is sensible from a music theory perspective by evaluating the learned chord embeddings. Surprisingly, our simple model managed to extract the circle of fifths, an important tool in music theory, from the dataset.

Introduction

Generating polyphonic music—where multiple notes are played simultaneously—presents a significant challenge for artificial intelligence. Traditional Recurrent Neural Networks (RNNs) often struggle to maintain long-term coherence, resulting in sequences that lack the structural "roadmap" found in human compositions. While some approaches rely on hard-coded music theory rules, JamBot explores whether a hierarchical Long Short-Term Memory (LSTM) architecture can autonomously learn these harmonic structures from raw data.

Methodology

The JamBot architecture employs a two-stage generation process designed to mimic the relationship between harmony and melody.

The JamBot architecture uses a hierarchical approach, splitting the task into chord progression prediction and polyphonic note generation.

The Two-Stage LSTM

  1. Chord LSTM: This network operates at the bar level. It treats chords as discrete units, similar to words in natural language processing. Using a 10-dimensional embedding layer, it predicts the next chord in a sequence, establishing the high-level harmonic structure.
  2. Polyphonic LSTM: This network generates the individual notes. For each of the eight time steps within a bar, it receives a vector containing the previous notes (piano roll), the current chord embedding, the next chord embedding (to anticipate transitions), and a binary counter indicating the position within the bar.

Data and Preprocessing

The model was trained on the Lakh MIDI Dataset, containing approximately 100,000 songs. To simplify the learning task, all songs were transposed to a common key (C major or A minor). Chords were automatically extracted by identifying the three most frequent notes within each bar.

Key Insights: Emergent Music Theory

The most significant finding of the JamBot project is the model's ability to "discover" fundamental music theory concepts without explicit instruction. By applying Principal Component Analysis (PCA) to the learned 10-dimensional chord embeddings, the researchers found that the model organized major chords into a structure mirroring the Circle of Fifths.

PCA visualization of learned chord embeddings. The model autonomously arranged the 12 major chords in a circular pattern corresponding to the Circle of Fifths.

This geometric relationship demonstrates that the model captured the underlying semantics of harmony. Chords that are musically related (sharing common notes or serving similar harmonic functions) are clustered together in the embedding space.

Results and Discussion

Qualitative evaluation indicates that JamBot produces music that is pleasing and largely free of jarring dissonance. By treating chords as a contextual "feature" rather than a rigid constraint, the Polyphonic LSTM is able to play passing tones that resolve naturally into the harmony. The inclusion of the "look-ahead" feature (the next chord embedding) was particularly effective in creating smooth, intentional transitions between bars.

Limitations and Future Directions

Despite its success, the model faces certain technical hurdles:

  • Note Duration: The current piano roll representation cannot distinguish between a single sustained note and multiple repeated notes.
  • Note Independence: Sampling notes independently can occasionally lead to awkward intervals. Future iterations could benefit from calculating joint probabilities for notes within a single time step.
  • Structural Depth: While the two-tier hierarchy improves long-term structure, additional layers may be necessary to capture complex musical forms like verse-chorus repetitions.

In conclusion, JamBot demonstrates that hierarchical LSTMs can bridge the gap between raw data and abstract musical concepts, providing a robust framework for creative AI in music generation.

Summary updated

Abstract

We propose a novel approach for the generation of polyphonic music based on LSTMs. We generate music in two steps. First, a chord LSTM predicts a chord progression based on a chord embedding. A second LSTM then generates polyphonic music from the predicted chord progression. The generated music sounds pleasing and harmonic, with only few dissonant notes. It has clear long-term structure that is similar to what a musician would play during a jam session. We show that our approach is sensible from a music theory perspective by evaluating the learned chord embeddings. Surprisingly, our simple model managed to extract the circle of fifths, an important tool in music theory, from the dataset.
BibTeX
@inproceedings{brunner2017jambot,
  author = {Gino Brunner and Yuyi Wang and Roger Wattenhofer and Jonas Wiesendanger},
  title = {{JamBot: Music Theory Aware Chord Based Generation of Polyphonic Music with LSTMs}},
  booktitle = {ICTAI},
  year = {2017},
  url = {https://openreview.net/forum?id=zrmcAqQzRO},
}
Figure