Abstract
A great number of deep learning based models have been recently proposed for automatic music composition. Among these models, the Transformer stands out as a prominent approach for generating expressive classical piano performance with a coherent structure of up to one minute. The model is powerful in that it learns abstractions of data on its own, without much human-imposed domain knowledge or constraints. In contrast with this general approach, this paper shows that Transformers can do even better for music modeling, when we improve the way a musical score is converted into the data fed to a Transformer model. In particular, we seek to impose a metrical structure in the input data, so that Transformers can be more easily aware of the beat-bar-phrase hierarchical structure in music. The new data representation maintains the flexibility of local tempo changes, and provides hurdles to control the rhythmic and harmonic structure of music. With this approach, we build a Pop Music Transformer that composes Pop piano music with better rhythmic structure than existing Transformer models.
Introduction
While deep learning models like the Music Transformer have succeeded in generating expressive classical music, they often struggle with the rigid structures of pop music. Pop music relies heavily on a steady, predictable pulse (the "beat") and a clear hierarchical structure consisting of beats, bars, and phrases.
Existing models typically use a "MIDI-like" representation that treats time as a series of absolute shifts (e.g., "wait 100ms"). Because the model must manually accumulate these shifts to identify the downbeat, it often loses track of the rhythm. This leads to "rhythmic drift," where the generated music feels unstable or off-kilter. The Pop Music Transformer addresses this by embedding musical domain knowledge—specifically metrical structure—directly into the data format using a representation called REMI (revamped MIDI-derived events).
Methods: The REMI Representation
The core innovation of this work is REMI, a new event-based representation for musical scores. Instead of absolute time-shifts, REMI introduces tokens that align with a metrical grid:
- Bar & Position: A
Bartoken marks the start of a measure, whilePositiontokens indicate where within the bar (e.g., 1/16th note increments) an event occurs. - Note Duration: Rather than using separate
Note-Offevents, which can be separated from theirNote-Oncounterparts by hundreds of tokens, REMI explicitly attaches aDurationto eachNote-On. - Tempo & Chord: Explicit tokens for local tempo changes and musical chords are introduced, enhancing controllability during the generation process.
The system utilizes Transformer-XL as the backbone architecture to maintain long-term dependencies. The training pipeline involves several Music Information Retrieval (MIR) techniques, including automatic transcription of audio piano covers into MIDI, beat tracking, and chord recognition. The model was trained on a dataset of 775 pop piano covers (approximately 48 hours of audio) specifically selected in 4/4 time.
Key Insights and Results
A primary finding is that "counting" is difficult for standard Transformers. By providing Bar and Position tokens, the model is given a metrical grid "for free," allowing it to focus on melody and harmony rather than basic timing. Furthermore, the use of Note Duration tokens helps the model learn the relationship between a note's pitch and its length more effectively than the standard Note-On/Note-Off paradigm.
The researchers evaluated the REMI-based model against several baselines, including a standard MIDI-like Transformer.
- Objective Metrics: The REMI model demonstrated a Beat/Downbeat Standard Deviation much closer to human training data, indicating superior rhythmic stability.
- Subjective Evaluation: In a listening test with 76 participants, the REMI model was consistently preferred, scoring 2.28 on a 3-point scale compared to 1.73 for the baseline.
- Qualitative Analysis: Visualizations of downbeat probability curves revealed that the REMI model develops a clear, periodic understanding of bar boundaries, whereas baselines often appear chaotic.
Conclusion
The Pop Music Transformer demonstrates that incorporating metrical structure into data representation significantly improves the quality of AI-generated music. While currently limited to 4/4 time and a 16th-note grid, this approach provides a robust foundation for future work. Expanding the model to handle complex time signatures, finer rhythmic quantization (like triplets), and multi-instrument arrangements remains an active area of research.
Summary updated