Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions

Yu-Siang Huang, Yi‐Hsuan Yang

TL;DR

This paper introduces REMI, a novel event representation for pop piano music that explicitly encodes metrical structure (bars and beats) and musical elements like tempo and chords. By feeding this richer representation into a Transformer-XL model, they achieve significantly more rhythmically coherent and subjectively preferred pop piano compositions compared to existing methods.

What Problem Does This Paper Address?

Generating expressive and structurally sound music with AI is a challenging task. While Transformer models have shown promise in generating coherent musical sequences, they often struggle with capturing the rhythmic regularity and hierarchical structure (beats, bars, phrases) inherent in music, especially genres like pop that rely on a strong beat. Existing approaches often represent music as a sequence of low-level events (like MIDI note on/off) that don't explicitly convey this metrical information, forcing the model to learn it implicitly, which can lead to rhythmic instability. This paper aims to bridge this gap by creating a more musically informed input representation for Transformer models.

Methods

The core innovation is the REMI (Revamped MIDI-derived Events) representation. Unlike traditional MIDI-like representations that rely on TIME-SHIFT events to infer timing, REMI introduces:

  • BAR and POSITION events: These explicitly mark the beginning of musical bars and specific quantized locations within a bar (e.g., 1/16th note resolution). This provides a clear metrical grid.
  • NOTE DURATION events: Replacing NOTE-OFF events, this makes note durations explicit, simplifying rhythm learning and avoiding issues with unpaired note-on/off events.
  • TEMPO events: These allow for local tempo changes, crucial for expressive performances.
  • CHORD events: These explicitly encode harmonic information, allowing for better control over chord progressions.

The authors use Transformer-XL as their backbone sequence model, chosen for its ability to handle long-range dependencies. The framework involves:

  1. Music Information Retrieval (MIR): Techniques like beat tracking, downbeat tracking, and chord recognition are used to extract the metrical and harmonic information needed to construct REMI events from audio or MIDI data.
  2. Sequence Modeling: The REMI event sequences are fed into the Transformer-XL model for training and generation.

Datasets

The authors collected 775 audio files of Pop piano music (approximately 48 hours) from the internet, including covers of anime, K-pop, and Western popular songs. They used the "Onsets and Frames" model for automatic piano transcription to convert these audio files into MIDI performances. The resulting transcribed MIDI files and their corresponding REMI event sequences are planned to be made publicly available.

Key Insights and Ideas

  • Explicit metrical structure is key: Simply providing raw note events to a Transformer is insufficient for generating rhythmically stable music. Explicitly encoding bars and beats (via BAR and POSITION events) significantly improves rhythmic coherence.
  • Richer event representations matter: Replacing NOTE-OFF with NOTE DURATION and adding TEMPO and CHORD events provides the model with more direct musical cues, leading to better performance and controllability.
  • Leveraging MIR for better AI music: Integrating established MIR techniques (beat tracking, chord recognition) into the data preparation pipeline is a powerful way to inject domain knowledge into generative models.
  • Transformer-XL is well-suited: Its ability to capture long-range dependencies is beneficial for musical structure, and it handles the REMI events effectively.

Key Results

  • Objective Evaluation: REMI-based models achieved significantly lower Beat STD and Downbeat STD (closer to the training data values), indicating more stable rhythms. REMI models also showed higher downbeat salience.
  • Subjective Evaluation: In a listening test with 76 participants, the REMI model was preferred over baselines (including an adapted Music Transformer) by a significant margin, especially by those with musical backgrounds.
  • Controllability: The paper demonstrates that by manipulating TEMPO and CHORD events, users can influence the generated music's tempo and harmonic content.

Limitations and Future Work

  • Rhythmic rigidity: The authors note that while REMI improves rhythmic stability, using a finer time grid (e.g., 32nd or 64th notes) to capture more nuanced rhythmic expression (like triplets or swing) led to fragmented and unpleasant music, suggesting limitations in the current Transformer-XL architecture for very fine-grained timing.
  • Multi-instrument generation: The current work focuses on solo piano. Future work aims to extend this to multi-instrument compositions (guitar, bass, drums).
  • Incorporating more musical information: The authors plan to include other high-level musical features like grooving and emotion.
  • Longer sequences: Exploring architectures better suited for modeling even longer musical sequences is suggested.
  • Conditioning on other modalities: Future work could involve conditioning music generation on lyrics or video.

Summary updated

Abstract

A great number of deep learning based models have been recently proposed for automatic music composition. Among these models, the Transformer stands out as a prominent approach for generating expressive classical piano performance with a coherent structure of up to one minute. The model is powerful in that it learns abstractions of data on its own, without much human-imposed domain knowledge or constraints. In contrast with this general approach, this paper shows that Transformers can do even better for music modeling, when we improve the way a musical score is converted into the data fed to a Transformer model. In particular, we seek to impose a metrical structure in the input data, so that Transformers can be more easily aware of the beat-bar-phrase hierarchical structure in music. The new data representation maintains the flexibility of local tempo changes, and provides hurdles to control the rhythmic and harmonic structure of music. With this approach, we build a Pop Music Transformer that composes Pop piano music with better rhythmic structure than existing Transformer models.
BibTeX
@inproceedings{huang2020pop,
  author = {Yu-Siang Huang and Yi‐Hsuan Yang},
  title = {{Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions}},
  booktitle = {ACM Multimedia},
  year = {2020},
}
Figure