Abstract
Modelling the real world complexity of music is a challenge for machine learning. We address the task of modeling melodic sequences from the same music genre. We perform a comparative analysis of two probabilistic models; a Dirichlet Variable Length Markov Model (Dirichlet-VMM) and a Time Convolutional Restricted Boltzmann Machine (TC-RBM). We show that the TC-RBM learns descriptive music features, such as underlying chords and typical melody transitions and dynamics. We assess the models for future prediction and compare their performance to a VMM, which is the current state of the art in melody generation. We show that both models perform significantly better than the VMM, with the Dirichlet-VMM marginally outperforming the TC-RBM. Finally, we evaluate the short order statistics of the models, using the Kullback-Leibler divergence between test sequences and model samples, and show that our proposed methods match the statistics of the music genre significantly better than the VMM.
Introduction
Modeling music is notoriously difficult for machine learning. Unlike text, where structure is often hierarchical and sequential, music relies on motifs—short melodic phrases—that repeat with variations such as transpositions, rhythmic shifts, or note substitutions at arbitrary points in time. Standard models like Variable Length Markov Models (VMMs) are often considered the state-of-the-art for melody generation, but they are "rigid," treating musical phrases as fixed strings of symbols. If a melody varies slightly, a standard VMM might fail to recognize it as the same motif. This research evaluates models that capture the "componential" nature of music—the idea that a melody is built from underlying features like scales, chords, and dynamics—while remaining flexible enough to handle the inherent variations of a specific genre.
Methodology
Two distinct architectures were evaluated to address the limitations of standard VMMs:
Dirichlet-VMM
This is a Bayesian upgrade to the standard VMM. While a normal VMM uses a Prediction Suffix Tree to track note sequences, it often struggles with sparsity (predicting notes it hasn't seen in a specific context before). The Dirichlet-VMM uses a hierarchical Dirichlet distribution for smoothing, allowing the model to share information across different levels of the tree and making it more robust when predicting rare or new sequences.
Time Convolutional RBM (TC-RBM)
Traditional Restricted Boltzmann Machines (RBMs) model data point-by-point. The TC-RBM instead applies convolution through time. It learns a set of filters (weights) that look at subsequences (e.g., 8 time-steps or one musical bar). Because it is convolutional, it possesses translation invariance—it can recognize a musical motif regardless of when it occurs in the piece.
Data Representation
The models were trained and tested on 117 traditional reels from the Nottingham Folk Music Database, consisting of Scottish and Irish dance tunes in G major with a 4/4 meter. Time was discretized into eighth-note intervals, and pitch was represented as a 1-of-26 vector covering two octaves, plus "silence" and a "continuation" marker to handle long notes.
Key Results and Insights
Latent Feature Discovery
The TC-RBM demonstrated an ability to "discover" musical concepts without prior instruction. By inspecting the learned weight filters, the authors found units that specifically responded to G-major scales, arpeggios, and specific chord progressions (like G-major vs. A-minor) in alternating time-steps.
Prediction Accuracy
Both the Dirichlet-VMM and TC-RBM significantly outperformed the standard VMM in log-likelihood tests for future note prediction. The Dirichlet-VMM had a slight edge in very short-term (next-step) prediction, likely due to the effectiveness of its Bayesian smoothing.
Statistical Matching
The authors used Kullback-Leibler (KL) divergence to assess how well the models' generated samples matched the statistics of the original genre. While both models were nearly perfect for 1st-order statistics (simple note frequencies), the TC-RBM was the winner for 2nd to 4th-order statistics (short sequences), capturing the "flavor" of the reels better than the tree-based models.
Conclusion
Both the Dirichlet-VMM and TC-RBM represent significant improvements over the previous state-of-the-art for melodic sequence modeling. While the Dirichlet-VMM excels at short-term prediction, the TC-RBM offers a more flexible, distributed representation that captures the stylistic nuances of a genre. Future work may involve stacking TC-RBMs into Deep Belief Networks to learn higher-level musical structures, such as the overall "A-B-A" form of a composition.
Summary updated