MMT-BERT: Chord-Aware Symbolic Music Generation Based on Multitrack Music Transformer and MusicBERT

Jinlong Zhu, Keigo Sakurai, Ren Togo, Takahiro Ogawa, Miki Haseyama

Introduction

Generating symbolic music, such as MIDI, that maintains long-term harmonic coherence remains a significant challenge in AI composition. Existing models often suffer from "harmonic ignorance," where individual notes are generated without a clear tonal center, leading to aimless melodies. Furthermore, managing the complexity of multitrack music—where multiple instruments must harmonize simultaneously—requires sophisticated architectural design.

MMT-BERT addresses these issues by introducing a "chord-aware" symbolic representation and a Generative Adversarial Network (GAN) framework. By explicitly teaching the model about underlying harmonies and leveraging a pre-trained music-understanding discriminator, MMT-BERT produces multitrack compositions that are more harmonious and human-like than previous state-of-the-art methods.

Chord-Aware Symbolic Representation

A core innovation of this work is the "quintuple token" representation. Unlike traditional MIDI event sequences, each musical event in MMT-BERT is defined by five distinct features: beat, position/degree, pitch/root, duration/mode, and instrument/extension.

To enforce harmonic structure, the researchers utilize the MusicLang chord analysis model to extract and insert specific Chord Events at the beginning of every bar. These tokens act as a harmonic blueprint, guiding the generator to choose notes that align with the intended tonal center and scale. This "scaffolding" approach ensures that the model does not have to guess the harmony from individual notes.

Quintuple token representation showing the integration of chord events.

The MMT-BERT Framework

The MMT-BERT architecture is structured as a GAN, consisting of a generator and a discriminator designed specifically for symbolic music.

Generator: Multitrack Music Transformer (MMT)

The generator is based on the Multitrack Music Transformer (MMT), which utilizes a Transformer-XL backbone. This choice is critical for handling long-range dependencies in music, allowing the model to maintain a "memory" of previous segments and ensure thematic consistency across a piece.

Discriminator: MusicBERT

The discriminator employs a fine-tuned MusicBERT, a model previously pre-trained on massive symbolic music datasets. Unlike standard discriminators trained from scratch, MusicBERT enters the GAN process with a pre-existing "understanding" of musical rules and structures. This allows it to provide more nuanced feedback to the generator, effectively acting as an expert critic.

Training Stability

To stabilize the GAN training process, the authors implement Relativistic Standard Loss. This loss function encourages the generator to produce data that is "more realistic" than the real samples, preventing the discriminator from becoming overwhelmingly powerful too early in the training process and ensuring a steady learning gradient.

The MMT-BERT GAN architecture featuring a Transformer-based generator and a pre-trained MusicBERT discriminator.

Experimental Evaluation

The model was trained on the Symbolic Orchestral Database (SOD), comprising 5,864 pieces. The dataset was partitioned into an 80/10/10 split for training, testing, and validation, with a focus on 4/4 time signatures to facilitate consistent chord extraction.

Quantitative Results

MMT-BERT demonstrated superior performance across several objective metrics compared to baselines like MMM and FIGARO:

  • Pitch Class Entropy Similarity (PCES): 99.73%, indicating high tonal stability.
  • Scale Consistency Similarity (SCS): 99.64%, confirming that generated notes adhere to the intended scales.
  • Groove Consistency Similarity (GCS): 99.66%, reflecting realistic rhythmic patterns.

Qualitative Results

Subjective listening tests involving 18 participants showed that MMT-BERT significantly outperformed the original MMT in categories of Richness and Humanness. While the model showed slight room for improvement in high-level "Structureness," the overall output was judged to be much closer to human-composed music.

Conclusion and Future Work

MMT-BERT successfully demonstrates that explicit harmonic scaffolding and pre-trained discriminators can significantly enhance the quality of AI-generated music. While the model excels at local harmony and multitrack coordination, future work will focus on improving global structures (such as A-B-A song forms) and addressing minor decoding gaps in the final audio conversion. The team also aims to explore multi-modal generation, using text or visual prompts to guide musical style.

Summary updated

Abstract

We propose a novel symbolic music representation and Generative Adversarial Network (GAN) framework specially designed for symbolic multitrack music generation. The main theme of symbolic music generation primarily encompasses the preprocessing of music data and the implementation of a deep learning framework. Current techniques dedicated to symbolic music generation generally encounter two significant challenges: training data's lack of information about chords and scales and the requirement of specially designed model architecture adapted to the unique format of symbolic music representation. In this paper, we solve the above problems by introducing new symbolic music representation with MusicLang chord analysis model. We propose our MMT-BERT architecture adapting to the representation. To build a robust multitrack music generator, we fine-tune a pre-trained MusicBERT model to serve as the discriminator, and incorporate relativistic standard loss. This approach, supported by the in-depth understanding of symbolic music encoded within MusicBERT, fortifies the consonance and humanity of music generated by our method. Experimental results demonstrate the effectiveness of our approach which strictly follows the state-of-the-art methods.
BibTeX
@inproceedings{zhu2024mmt,
  author = {Jinlong Zhu and Keigo Sakurai and Ren Togo and Takahiro Ogawa and Miki Haseyama},
  title = {{MMT-BERT: Chord-Aware Symbolic Music Generation Based on Multitrack Music Transformer and MusicBERT}},
  booktitle = {ISMIR},
  year = {2024},
  url = {https://openreview.net/forum?id=oc3ycVGdqK},
}

Loading related content...

Figure