Abstract
Piano covers of pop music are enjoyed by many people. However, the task of automatically generating piano covers of pop music is still understudied. This is partly due to the lack of synchronized {Pop, Piano Cover} data pairs, which made it challenging to apply the latest data-intensive deep learning-based methods. To leverage the power of the data-driven approach, we make a large amount of paired and synchronized {Pop, Piano Cover} data using an automated pipeline. In this paper, we present Pop2Piano, a Transformer network that generates piano covers given waveforms of pop music. To the best of our knowledge, this is the first model to generate a piano cover directly from pop audio without using melody and chord extraction modules. We show that Pop2Piano, trained with our dataset, is capable of producing plausible piano covers.
Introduction
Generating a piano cover of a pop song is a creative act of "music reduction" that requires capturing the harmony, rhythm, and atmosphere of the original while adapting it to the piano's unique textures. Traditionally, automated systems have relied on complex modular pipelines—extracting melodies and chords first before applying rule-based arrangement scripts. These systems often fail to capture the implicit characteristics of a song, such as the specific energy of a bridge or the unique style of a human arranger.
Pop2Piano addresses these limitations by being the first model to generate piano covers directly from raw audio waveforms. By treating cover generation as an end-to-end sequence-to-sequence task, the model learns the direct mapping from audio to arrangement.
Data Acquisition: The PSP Dataset
The primary bottleneck in training deep learning models for this task is the scarcity of synchronized paired data. While pop songs and their piano covers are abundant online, they are rarely aligned in time or key. To solve this, the authors developed an automated synchronization pipeline to create the Piano Cover Synchronized to Pop Audio (PSP) dataset.
The pipeline involves transcribing human piano covers into MIDI, using Dynamic Time Warping (DTW) to align the MIDI with the original pop audio, and quantizing timings to 8th-note beats. This resulted in a 300-hour dataset derived from 21 different YouTube arrangers.
Methodology
Pop2Piano utilizes a Transformer-based architecture, specifically a T5-small variant, to map audio features to MIDI events.
Architecture and Style Conditioning
The model takes a log-Mel spectrogram of the pop audio as input. A key innovation is the use of Arranger Tokens. Since different arrangers have distinct styles (e.g., varying note densities or harmonic complexity), the model is conditioned on a specific "Arranger ID." This allows the model to mimic different arrangement styles during generation.
The decoder generates MIDI events as tokens, including Note Pitch, Note On/Off, and "Beat Shifts" for relative timing. This end-to-end approach bypasses the need for explicit melody or chord extraction modules.
Results and Discussion
The model demonstrates that explicit musical knowledge is not strictly necessary for generating coherent arrangements if provided with sufficient synchronized data.
- Style Control: The model showed a high correlation ($R^2 = 0.804$) between the note density of generated covers and the target arranger's style, proving the effectiveness of Arranger Tokens.
- Subjective Evaluation: In user studies, 70% of participants preferred Pop2Piano's outputs over a baseline model trained on the POP909 dataset.
- Performance: Pop2Piano achieved a Mean Opinion Score (MOS) of 3.216. While still below human ground truth (3.771), it marks a significant advancement in audio-to-MIDI generation.
Conclusion and Future Work
Pop2Piano represents a major step toward automated, stylistically varied music arrangement. However, limitations remain, such as a short context window (4-beat segments) which can affect long-term structural consistency. Additionally, the current 8th-note quantization prevents the generation of complex rhythms like triplets or 16th-note runs. Future work will focus on expanding the context window and improving rhythmic resolution to capture more intricate performance nuances.
Summary updated