Unsupervised Transcription of Piano Music

Taylor Berg-Kirkpatrick, Jacob Andreas, Dan KleinNeurIPS 2014 · OpenReview

Automatic Music Transcription (AMT)—the process of converting raw audio into symbolic notation like MIDI—is often considered the "holy grail" of music information retrieval. Piano music presents a particularly steep challenge for two primary reasons. First is timbre variability: every piano sounds different, and a single note is a complex "envelope" of harmonics that changes based on the specific instrument and room acoustics. Second is polyphony and collisions: when multiple notes are played simultaneously, their harmonics overlap. Because musical consonance inherently involves overlapping harmonics, it is mathematically difficult to disentangle which note produced which sound.

Existing methods typically force a choice: either model the musical structure (the "grammar" of the music) or adapt to the instrument's specific timbre. Combining both is computationally expensive because the dependencies between notes break the fast algorithms usually used for transcription. To solve this, researchers from UC Berkeley have developed a new generative probabilistic model for unsupervised piano transcription that jointly learns instrument-specific timbre and musical structure.

The Generative Model

The authors propose a generative model that mimics the causal, physical process of a piano producing sound. Instead of relying on a single, rigid algorithm, the system is broken down into three interconnected components:

  1. Event Model ($M$): A hidden semi-Markov model represents the symbolic "score." It dictates whether a note is being played, its duration, and its velocity (how hard the key is struck).
  2. Activation Model ($A$): This represents the "envelope" or loudness of a string over time, capturing the sharp attack when a hammer hits and the gradual decay as the string vibrates.
  3. Component Spectrogram Model ($S$): This models the "spectral profile"—the specific recipe of harmonics—for each of the 88 keys on the piano.
Overview of the generative probabilistic model, illustrating how event, envelope, and spectral parameters combine to form the final audio spectrogram.

Crucially, the model uses Poisson noise to represent the final spectrogram. This aligns the work with Non-negative Matrix Factorization (NMF) techniques but injects a layer of musical "priors" that standard NMF lacks.

Modeling Physical Sound Production

A key insight of this paper is embedding physical intuition into the mathematical model. By explicitly modeling the acoustic "envelope" (the rise and fall of a note), the system naturally accounts for the fact that a note sounds fundamentally different at its initial attack than it does three seconds into its decay. Many simple frequency-based models miss this temporal nuance.

Detailed view of the Event and Activation models, showing how note duration, velocity, and envelope parameters generate the physical acoustic envelope over time.

Inference and Joint Learning

The "magic" of this approach happens during the inference step. To handle the computational complexity of joint modeling, the authors employ block-coordinate ascent (Iterated Conditional Modes). The system alternates between two steps:

  • Updating the transcription (the notes) using a modified Viterbi-like algorithm.
  • Updating the instrument's timbre (spectral profiles and envelopes) using gradient-based optimization.

This proves that you don't have to choose between modeling "what is being played" and "what the instrument sounds like." By iterating, the model uses its current transcription to better learn the timbre, and uses its refined understanding of the timbre to improve the transcription.

Datasets and Key Results

The model was evaluated on the MAPS (MIDI-Aligned Piano Sounds) dataset, a standard corpus of high-quality recordings from an acoustic "Disklavier" piano that provides ground-truth MIDI data. The system was tested in a strictly unsupervised setting, meaning it had never heard the specific test piano before.

The results demonstrated significant improvements over existing methods:

  • Onset Accuracy: The model achieved an Onset $F_1$ score of 76.4, representing a 10.6% relative gain over the previous best unsupervised system.
  • Frame-level Accuracy: It achieved an $F_1$ of 74.4, outperforming previous state-of-the-art frame-level metrics.
  • Beating Supervised Baselines: Surprisingly, this unsupervised model outperformed a state-of-the-art supervised discriminative baseline (which scored 70.4). This is likely because the unsupervised model's ability to adapt to the specific acoustics of the test recording gave it a distinct advantage over a rigid, pre-trained supervised model.

The authors noted that timbral adaptation is critical; when they skipped the step of adapting to the specific piano's timbre, the Onset $F_1$ dropped from 76.4 to 72.6, proving that "one-size-fits-all" acoustic models are fundamentally limited.

Limitations and Future Work

While highly effective, the model has a few known limitations. It occasionally suffers from "octave errors," confusing a note with one an octave higher or lower due to near-perfect harmonic overlap. It also sometimes exhibits segmentation errors, transcribing a single long-held note as several short, repeated notes. Finally, while block-coordinate ascent makes the math tractable, it remains more computationally intensive than simple signal-processing heuristics.

Future directions include incorporating more nuanced musical structures—such as explicitly modeling rhythm or harmony—and expanding the timbral models to handle different types of instruments beyond the piano.

Summary updated

Abstract

We present a new probabilistic model for transcribing piano music from audio to a symbolic form. Our model reflects the process by which discrete musical events give rise to acoustic signals that are then superimposed to produce the observed data. As a result, the inference procedure for our model naturally resolves the source separation problem introduced by the the piano's polyphony. In order to adapt to the properties of a new instrument or acoustic environment being transcribed, we learn recording specific spectral profiles and temporal envelopes in an unsupervised fashion. Our system outperforms the best published approaches on a standard piano transcription task, achieving a 10.6% relative gain in note onset F1 on real piano audio.
BibTeX
@inproceedings{bergkirkpatrick2014unsupervised,
  author = {Taylor Berg-Kirkpatrick and Jacob Andreas and Dan Klein},
  title = {{Unsupervised Transcription of Piano Music}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2014},
  url = {https://openreview.net/forum?id=rJELV_WOZr},
}
Figure