Pop2Piano : Pop Audio-Based Piano Cover Generation

Jongho Choi, Kyogu Lee

Generating a piano cover of a pop song is more than just transcribing a melody; it’s a creative act of "music reduction" that requires capturing the harmony, rhythm, and "vibe" of the original while adapting it to the unique textures of the piano. Traditionally, automated systems have relied on complex pipelines—extracting melodies and chords first before applying rule-based arrangement scripts.

In this paper, researchers from Rebellions Inc. and Seoul National University present Pop2Piano, the first model to generate piano covers directly from raw audio waveforms using a Transformer-based architecture.

TL;DR

Pop2Piano is an end-to-end Transformer model that generates MIDI piano covers directly from pop audio spectrograms. By introducing a new 300-hour synchronized dataset (PSP) and using "arranger tokens" to control musical style, the model produces plausible, stylistically varied covers without needing intermediate melody or chord extraction modules.

What Problem Does This Paper Address?

The primary challenge in automatic piano cover generation has been the lack of synchronized paired data. While there are plenty of pop songs and plenty of piano covers on the internet, they are rarely aligned in time or key, making it difficult to train deep learning models that require direct input-output mapping.

Previous attempts often bypassed this by using a "modular" approach:

  1. Extract the melody.
  2. Extract the chords.
  3. Use a rule-based system to generate piano notes.

The downside? These systems often miss the "implicit" characteristics of a song—the atmosphere, the specific energy of a bridge, or the unique style of a human arranger. Pop2Piano aims to skip the middleman and learn the mapping from audio to arrangement directly.

Methods

The authors treat cover generation as a sequence-to-sequence task, similar to machine translation or Automatic Music Transcription (AMT).

1. The Architecture

The model uses a T5-small (Transformer) architecture.

  • Input: The model takes a log-Mel spectrogram of the pop audio.
  • Style Conditioning: A unique feature is the Arranger Token. Because different people arrange music differently (some use dense chords, others prefer simple melodies), the authors prepended a token representing a specific arranger to the input. This allows the model to "mimic" different arrangement styles.
  • Output: The model generates MIDI events as tokens, including Note Pitch, Note On/Off, and "Beat Shifts" (relative time jumps).

2. The Synchronization Pipeline

To create their training data, the authors developed a sophisticated preprocessing pipeline:

  • Transcription: Human piano covers (audio) were converted to MIDI.
  • Alignment: They used Dynamic Time Warping (DTW) to align the piano MIDI with the original pop audio.
  • Quantization: Timings were snapped to 8th-note beats to reduce data "noise" and make the learning task more manageable.

Datasets

  • PSP (Piano Cover Synchronized to Pop Audio): The authors' primary contribution to the community. It consists of 300 hours of synchronized {Pop, Piano Cover} pairs derived from 21 different arrangers on YouTube.
  • POP909: A well-known dataset of 909 Chinese pop songs with MIDI arrangements, used here as a baseline for comparison.

Key Insights and Ideas

  • End-to-End is Viable: The model proves that explicit musical knowledge (like "this is a C-major chord") isn't strictly necessary for a model to generate musically coherent arrangements if the dataset is large and synchronized enough.
  • Arranger Tokens Work: By conditioning the model on specific arrangers, the researchers found they could control the note density of the output. If you want a "busy" cover, you use a token from an arranger known for complex pieces.
  • Bypassing the "Correct Answer" Problem: Unlike transcription (where there is one right note), cover generation is subjective. The authors addressed this by focusing on "plausibility" and style consistency rather than bit-for-bit accuracy to a ground truth.

Key Results

  • Style Mimicry: The model showed a high correlation (R² = 0.804) between the note density of the generated covers and the target arranger's actual style.
  • Subjective Quality: In a user study, 70% of participants preferred Pop2Piano's outputs over a baseline model trained on the POP909 dataset.
  • Mean Opinion Score (MOS): On a scale of 1-5, Pop2Piano achieved a score of 3.216. While lower than human ground truth (3.771), it represents a significant step forward for end-to-end audio-to-MIDI generation.

Limitations and Future Work

  • Short Context Window: The model currently processes audio in 4-beat segments. This can lead to a lack of long-term structural consistency (e.g., the model might "forget" how it handled the first chorus when it gets to the second).
  • Rhythmic Constraints: Due to 8th-note quantization, the model cannot currently generate complex rhythms like triplets, 16th-note runs, or trills.
  • Audio Quality: The generated MIDI is only as good as the piano synthesizer used to play it back; the paper focuses on note generation rather than the "performance" aspect (velocity, pedaling).

The authors have released a demo on Colab for those interested in hearing the results for themselves.

Summary updated

Abstract

Piano covers of pop music are enjoyed by many people. However, the task of automatically generating piano covers of pop music is still understudied. This is partly due to the lack of synchronized {Pop, Piano Cover} data pairs, which made it challenging to apply the latest data-intensive deep learning-based methods. To leverage the power of the data-driven approach, we make a large amount of paired and synchronized {Pop, Piano Cover } data using an automated pipeline. In this paper, we present Pop2Piano, a Transformer network that generates piano covers given waveforms of pop music. To the best of our knowledge, this is the first model to generate a piano cover directly from pop audio without using melody and chord extraction modules. We show that Pop2Piano, trained with our dataset, is capable of producing plausible piano covers.
BibTeX
@inproceedings{choi2023pop,
  author = {Jongho Choi and Kyogu Lee},
  title = {{Pop2Piano : Pop Audio-Based Piano Cover Generation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=IsR56GvxI2},
}

Loading related content...

Figure