Discrete Diffusion Probabilistic Models for Symbolic Music Generation

Matthias Plasser, Silvan Peter, Gerhard WidmerIJCAI 2023 · DOI

TL;DR

This paper introduces SCHUBERT, a novel discrete diffusion probabilistic model (D3PM) for generating symbolic music. It achieves state-of-the-art sample quality and offers unprecedented flexibility for tasks like music infilling and accompaniment generation, all while being significantly more parameter-efficient than previous methods.

What Problem Does This Paper Address?

Generating high-quality symbolic music has been a long-standing challenge in AI. While diffusion models have shown great success in continuous domains like audio, their application to discrete symbolic music representations (like MIDI) has been limited. Existing approaches often rely on indirect methods, such as modeling continuous latent spaces of symbolic representations, which can be less direct and efficient. Furthermore, evaluating the quality of generated music is notoriously difficult, with statistical metrics sometimes failing to capture true musicality and being susceptible to manipulation. This paper aims to bridge this gap by directly applying discrete diffusion models to symbolic music generation and critically examining current evaluation practices.

Methods

The core of this work is the SCHUBERT model, which leverages an Absorbing State Diffusion Probabilistic Model (ASD3PM). Unlike standard diffusion models that gradually add noise, ASD3PMs introduce a special "absorbing state" (akin to a [MASK] token in language models) to represent corrupted data. This allows for direct, non-autoregressive generation and infilling of symbolic music tokens.

Key methodological contributions include:

  • Direct Discrete Modeling: Applying D3PMs directly to symbolic music tokens, avoiding the need for intermediate continuous representations.
  • Absorbing State D3PM (ASD3PM): Utilizing a specific D3PM formulation that simplifies training and enables flexible infilling.
  • Non-Autoregressive Generation: Enabling parallel processing of tokens, leading to faster generation and more flexible control over the output.
  • Hierarchical Architecture: Employing a novel architecture with lower and upper hierarchy levels (using 1D CNNs and Transformer layers) to efficiently process long musical sequences.
  • Post-hoc Classifier Guidance: Demonstrating that the model can be guided by external classifiers (e.g., for note density) without retraining, offering a new avenue for user control.

Datasets

The models were trained and evaluated on the Lakh MIDI dataset (LMD), a large collection of over 170,000 multi-track MIDI files. For direct comparison with a prior work, a subset of 1,000,000 melodies was used without augmentation. The evaluation also involved comparing generated pieces against 1,000 pieces drawn from the joined train and evaluation set as ground truth.

Key Insights and Ideas

  • Discrete Diffusion is Viable for Music: The paper convincingly demonstrates that discrete diffusion models can directly and effectively generate high-quality symbolic music, opening up new possibilities beyond continuous domain models.
  • Parameter Efficiency Matters: SCHUBERT achieves state-of-the-art results with significantly fewer trainable parameters (80M) compared to its competitor (425M), suggesting a more efficient approach to symbolic music generation.
  • Infilling and Control are Powerful: The non-autoregressive nature and the ASD3PM formulation enable flexible infilling (filling in masked musical passages) and accompaniment generation, offering greater creative control.
  • Statistical Metrics Can Be Deceiving: A critical insight is the paper's demonstration that standard statistical metrics for music generation can be "gamed" by creating non-musical samples that still score highly. This highlights the need for more robust evaluation methods beyond simple statistical aggregation.
  • Post-hoc Guidance is Promising: The ability to guide generation with external classifiers (like note density) without retraining the diffusion model itself is a significant step towards more interactive and controllable music generation systems.

Key Results

  • State-of-the-Art Sample Quality: SCHUBERT consistently outperforms the reference model (Mittal et al., 2021) on framewise self-similarity metrics for both melody and trio generation.
  • Improved Parameter Efficiency: SCHUBERT has approximately 5 times fewer trainable parameters than the reference model while achieving better or comparable results.
  • Flexible Infilling: The model demonstrates strong performance in infilling tasks, successfully reconstructing masked musical passages in melody, bass, and drum tracks.
  • Classifier Guidance Success: A proof-of-concept experiment showed that guiding the model to achieve specific note densities per measure resulted in high accuracy with only a moderate loss in subjective sample quality.

Limitations and Future Work

  • Limitations of Statistical Metrics: The authors critically highlight the limitations of current statistical metrics, demonstrating how they can be confounded by non-musical samples. They suggest that while their model performs well on these metrics, the metrics themselves may not fully capture true musical quality.
  • Subjective Evaluation Needed: While quantitative results are strong, the paper acknowledges that subjective listening tests are still crucial for a complete evaluation of musical quality.
  • Future Work: The authors suggest exploring more sophisticated classifier guidance techniques, investigating the impact of different masking schedules for trio generation, and further exploring the interactive capabilities of their model. They also implicitly point towards the need for developing better evaluation metrics that are less susceptible to manipulation and better reflect musical concepts.

Summary updated

BibTeX
@inproceedings{plasser2023discrete,
  author = {Matthias Plasser and Silvan Peter and Gerhard Widmer},
  title = {{Discrete Diffusion Probabilistic Models for Symbolic Music Generation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/648},
}
Figure