JamBot: Music Theory Aware Chord Based Generation of Polyphonic Music with LSTMs

Gino Brunner, Yuyi Wang, Roger Wattenhofer, Jonas Wiesendanger

JamBot: How LSTMs "Discover" the Circle of Fifths

Generating music with artificial intelligence often feels like a balancing act between chaos and rigidity. Early models either produced aimless, dissonant melodies or were so constrained by hard-coded music theory rules that they lacked creativity. JamBot, a research project from ETH Zürich, proposes a middle ground: a two-stage LSTM architecture that uses chords as a structural guide while allowing the model to "discover" the underlying laws of harmony on its own.

TL;DR

JamBot uses a hierarchical LSTM approach to generate polyphonic music by first predicting a chord progression and then using those chords to guide note generation. The most striking result is that the model autonomously learns fundamental music theory concepts, like the Circle of Fifths, directly from raw MIDI data without any explicit instruction.

What Problem Does This Paper Address?

Generating polyphonic music (multiple notes played simultaneously) is significantly harder than generating a single melody line. Recurrent Neural Networks (RNNs) often struggle with "long-term structure"—the ability to make a piece of music feel like a coherent journey rather than a random sequence of notes.

While some researchers have tried to solve this by hard-coding music theory into their models, the authors of JamBot wanted to see if a model could learn these structures from the data itself. They aimed to create a system that produces "harmonic sounding" music that mimics the feel of a human musician improvising during a jam session.

Methods

The JamBot architecture is split into two distinct Long Short-Term Memory (LSTM) networks:

  1. The Chord LSTM: This network treats chords like words in a sentence. It uses a 10-dimensional embedding layer (similar to Word2Vec in NLP) to represent chords. It predicts the next chord in a sequence, providing the high-level "roadmap" for the piece.
  2. The Polyphonic LSTM: This network generates the actual notes. At each of the eight time steps within a musical bar, it receives a concatenated input vector consisting of:
  3. The piano roll (the notes played in the previous step).
  4. The embedding of the current chord.
  5. The embedding of the next chord (allowing the model to "lead" into transitions).
  6. A binary counter (0-7) that tells the model where it is within the current bar.

To prevent the output from becoming too "cluttered" (playing too many notes at once), the authors implemented a soft upper limit that scales down note probabilities if the sum of all predicted notes exceeds a certain threshold.

Datasets

  • Lakh MIDI Dataset: A massive collection of approximately 100,000 songs in MIDI format.
  • Preprocessing: The authors filtered the dataset for songs in major or natural minor scales and transposed everything to a single key (C major/A minor). This "shifted dataset" allowed the model to focus on learning harmonic patterns rather than memorizing twelve different versions of the same scale.
  • Chord Extraction: Since MIDI files don't always explicitly label chords, the authors developed an automated process to extract the three most frequent notes in a bar to define the underlying chord.

Key Insights and Ideas

  • Emergent Music Theory: The "aha moment" of the paper occurs when the authors visualize the learned chord embeddings using Principal Component Analysis (PCA). When they plotted the embeddings of the 12 major chords, the points formed a near-perfect Circle of Fifths. The model didn't just learn which chords follow each other; it learned the geometric relationship between keys that musicians have used for centuries.
  • Chords as Context, Not Rules: Unlike models that force the output to only use notes from a specific chord, JamBot's Polyphonic LSTM treats the chord as a "feature." This allows it to play "passing tones" or dissonant notes that resolve into the harmony, making the music sound more natural and less mechanical.
  • The Power of "Looking Ahead": By feeding the next chord into the Polyphonic LSTM, the model learns to anticipate harmonic changes, creating smoother transitions that feel intentional rather than abrupt.

Key Results

  • Qualitative Harmony: The generated music is described as sounding pleasing and harmonic with very few dissonant notes. It successfully maintains a clear long-term structure over multiple bars.
  • Embedding Accuracy: The model's ability to cluster related chords (those sharing common notes) in the embedding space demonstrates that it captured the "semantics" of music.
  • Efficiency: Because it works with MIDI data rather than raw audio waveforms, JamBot is relatively fast to train, requiring only a single GPU.

Limitations and Future Work

  • The "Held Note" Problem: The standard piano roll representation used here cannot distinguish between one long note held for four beats and four short notes played in succession.
  • Note Independence: Currently, the model samples each note's probability independently. This can occasionally lead to weird intervals. The authors suggest moving toward calculating joint probabilities to ensure that the notes played at a single time step are more harmonically related.
  • Higher-Level Hierarchy: While the two-step process helps with structure, the authors want to add even more layers to the hierarchy to handle larger musical structures like repeating verses and choruses.

Summary updated

Abstract

We propose a novel approach for the generation of polyphonic music based on LSTMs. We generate music in two steps. First, a chord LSTM predicts a chord progression based on a chord embedding. A second LSTM then generates polyphonic music from the predicted chord progression. The generated music sounds pleasing and harmonic, with only few dissonant notes. It has clear long-term structure that is similar to what a musician would play during a jam session. We show that our approach is sensible from a music theory perspective by evaluating the learned chord embeddings. Surprisingly, our simple model managed to extract the circle of fifths, an important tool in music theory, from the dataset.
BibTeX
@inproceedings{brunner2017jambot,
  author = {Gino Brunner and Yuyi Wang and Roger Wattenhofer and Jonas Wiesendanger},
  title = {{JamBot: Music Theory Aware Chord Based Generation of Polyphonic Music with LSTMs}},
  booktitle = {ICTAI},
  year = {2017},
  url = {https://openreview.net/forum?id=zrmcAqQzRO},
}

Loading related content...

Figure