The Effect of Explicit Structure Encoding of Deep Neural Networks for Symbolic Music Generation

Ke Chen, Weilin Zhang, Shlomo Dubnov, Gus XiaOpenReview 2018 · DOI · OpenReview

Abstract

With recent breakthroughs in artificial neural networks, deep generative models have become one of the leading techniques for computational creativity. Despite very promising progress on image and short sequence generation, symbolic music generation remains a challenging problem since the structure of compositions are usually complicated. In this study, we attempt to solve the melody generation problem constrained by the given chord progression. In particular, we explore the effect of explicit architectural encoding of musical structure via comparing two sequential generative models: LSTM (a type of RNN) and WaveNet (dilated temporal-CNN). As far as we know, this is the first study of applying WaveNet to symbolic music generation, as well as the first systematic comparison between temporal-CNN and RNN for music generation. We conduct a survey for evaluation in our generations and implemented Variable Markov Oracle in music pattern discovery. Experimental results show that to encode structure more explicitly using a stack of dilated convolution layers improved the performance significantly, and a global encoding of underlying chord progression into the generation procedure gains even more.

Introduction

Generating musically coherent melodies is a significant challenge for artificial intelligence. Unlike simple sequences, music is defined by hierarchical structures, long-term dependencies, and motifs that must align with a global harmonic framework. While models have succeeded in mimicking local textures, long-term coherence often remains elusive. This research investigates whether musical structure is better learned implicitly through internal memory (LSTMs) or if it should be "baked into" the network architecture itself (WaveNet).

Methodology

The researchers framed the task as conditioned melody generation, where a melody is generated to fit a pre-defined chord progression. Three primary architectures were evaluated:

  1. Unidirectional LSTM: A standard recurrent model that processes notes sequentially, with access only to current and past chord information.
  2. Bidirectional LSTM (Bi-LSTM): An advanced recurrent architecture that processes the chord progression in both directions, allowing the model to "look ahead" at future harmonic goals.
  3. WaveNet (Dilated Temporal-CNN): Adapted from raw audio generation to symbolic MIDI-like data. WaveNet utilizes dilated convolutions to provide a massive receptive field, enabling the model to observe a wide window of history without excessive computational cost.

Data Representation

To handle the complexities of pitch and rhythm, the authors employed a "staggered representation." This encoding accounts for 130 dimensions for melody (including pitch, rest states, and holding states to distinguish sustained notes) and 25 dimensions for chord information.

Figure 1: The data representation maps piano scores to a high-dimensional matrix, encoding melody and chord states separately to preserve rhythmic and harmonic nuances.

Experimental Setup

The models were trained on the Nottingham Database, consisting of 941 folk songs. Data was augmented via transposition into all 12 keys and normalized to a fixed 120 BPM with a 1/16th note frame size. Evaluation was conducted through two lenses: a subjective human survey of 106 participants and an objective pattern discovery analysis using the Variable Markov Oracle (VMO) algorithm.

Key Insights and Results

The core finding of the study is the superiority of explicit structural encoding. WaveNet’s architecture naturally mirrors musical hierarchy: its layers of dilated convolutions capture relationships across expanding time scales (e.g., 1/16 notes to 1/8 notes). This allows for superior rhythmic mastery, including the generation of complex patterns like triplets.

Figure 2: The deep Res-Bi-LSTM architecture used in the study, which leverages bidirectional processing to incorporate global chord constraints into the generation process.

However, the Bi-LSTM emerged as the overall winner in human preference. The researchers attribute this to the "teleological" nature of music—it moves toward a goal. By encoding the entire chord progression as a global constraint, the Bi-LSTM generates notes that resolve satisfyingly into upcoming chord changes. Objective VMO analysis confirmed these results, showing that both Bi-LSTM and WaveNet produced significantly more stable motifs than the standard Unidirectional LSTM. Statistical significance was robust, with ANOVA tests yielding p-values as low as $3.02e^{-16}$.

Conclusion

This study demonstrates that the way we encode musical time and global constraints significantly impacts the coherence of AI-generated music. While WaveNet excels at hierarchical rhythmic structures, the Bi-LSTM's ability to "see the future" of a chord progression makes it uniquely suited for harmonic alignment. Future work may explore "Bidirectional WaveNet" architectures to combine the strengths of both approaches.

Summary updated

Abstract

With recent breakthroughs in artificial neural networks, deep generative models have become one of the leading techniques for computational creativity. Despite very promising progress on image and short sequence generation, symbolic music generation remains a challenging problem since the structure of compositions are usually complicated. In this study, we attempt to solve the melody generation problem constrained by the given chord progression. This music meta-creation problem can also be incorporated into a plan recognition system with user inputs and predictive structural outputs. In particular, we explore the effect of explicit architectural encoding of musical structure via comparing two sequential generative models: LSTM (a type of RNN) and WaveNet (dilated temporal-CNN). As far as we know, this is the first study of applying WaveNet to symbolic music generation, as well as the first systematic comparison between temporal-CNN and RNN for music generation. We conduct a survey for evaluation in our generations and implemented Variable Markov Oracle in music pattern discovery. Experimental results show that to encode structure more explicitly using a stack of dilated convolution layers improved the performance significantly, and a global encoding of underlying chord progression into the generation procedure gains even more.
BibTeX
@misc{chen2018effect,
  author = {Ke Chen and Weilin Zhang and Shlomo Dubnov and Gus Xia},
  title = {{The Effect of Explicit Structure Encoding of Deep Neural Networks for Symbolic Music Generation}},
  year = {2018},
  eprint = {1811.08380},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.08380},
  howpublished = {OpenReview},
}
Figure