Automatic Music Transcription and Audio Source Separation

Mark D. Plumbley, Samer A. Abdallah, Juan Pablo Bello, Mike E. Davies, Giuliano Monti, Mark B. SandlerCybern. Syst 2002 · DOI

TL;DR

This paper explores methods for automatically transcribing music into a score and separating mixed audio sources. It presents a range of techniques, from autocorrelation for monophonic transcription to blackboard models and multiple-cause/sparse coding for polyphonic music, and discusses independent component analysis (ICA) for audio source separation. The work highlights the challenges and potential of these approaches for understanding and manipulating musical audio.

What Problem Does This Paper Address?

The paper tackles two intertwined problems in computational auditory scene analysis: automatic music transcription and audio source separation.

Automatic music transcription aims to convert an audio recording of music into a written musical score, detailing the instruments playing, the notes, and their durations. This is a complex task because music often involves multiple instruments playing simultaneously (polyphony), with overlapping frequencies and timbres. Even for humans, perfect musical transcription is a skill that requires significant training and is limited in complexity.

Audio source separation, often referred to as the "cocktail party problem," deals with disentangling individual sound sources from a mixed audio signal, especially when multiple microphones are available. This is crucial for isolating specific instruments or voices in a recording.

The paper bridges these two problems by suggesting that source separation techniques can enhance music transcription by providing information about the spatial location of instruments, and that understanding the structure of musical notes (their spectral shapes) is key to both transcription and separation.

Methods

The paper outlines several approaches:

  • Monophonic Music Transcription:
    • Autocorrelation-based method: This technique analyzes the periodicity of the audio signal to identify the fundamental frequency (pitch) of a single note. It involves calculating the autocorrelation function of the signal and finding peaks that correspond to the waveform's period.
  • Onset detection: Used in conjunction with pitch tracking, this method identifies the start of new notes by looking for sharp changes in the high-frequency content of the audio signal.
  • MIDI conversion: The identified pitches and durations are then converted into a MIDI (Musical Instrument Digital Interface) format.
  • Polyphonic Music Transcription:
    • Blackboard System: This is a knowledge-based approach where different "knowledge sources" (experts) contribute hypotheses about the music (e.g., frequency tracks, partials, notes) to a shared "blackboard" (hypothesis database). A scheduler manages the interaction of these experts to build a coherent transcription.
  • Multiple-Cause Model: This is a data-driven approach, often using neural networks, that attempts to model the underlying "causes" (notes and instruments) that generate the observed audio signal. It learns the spectral "shapes" of notes and instruments and their relative contributions to the mix.
  • Sparse Coding: This method leverages the observation that in polyphonic music, only a few notes are typically active at any given time. It aims to find a representation where most values are zero, making it efficient for identifying note activations and their spectral characteristics.
  • Audio Source Separation:
    • Independent Component Analysis (ICA): This is a statistical method used to separate a multivariate signal into additive subcomponents assuming that the subcomponents are non-Gaussian and mutually independent. The paper describes a typical two-stage ICA process: whitening (decorrelation) and then rotating the axes to achieve independence.
  • Convolutive Mixing: The paper acknowledges that real-world audio mixing often involves delays and echoes, leading to convolutive mixtures. It proposes an approach that models the variation of frequency band amplitudes for each source and uses a likelihood ratio jump test to determine the correct permutation of separated sources.

Datasets

The paper mentions the use of:

  • Synthesized sounds: Artificial spectra from synthesized clarinet, violin, and alto recorder notes were used to test the multiple-cause model.
  • University of Iowa musical instrument samples: Real audio signals were constructed by adding pulses of notes from various instruments from this collection.
  • Bach: Partita in A minor for keyboard (BWV827): This piece was used to test the sparse coding algorithm on real MIDI-generated sounds.
  • Speech sources: Used to compare the proposed source separation algorithm with the Smaragdis (1997) algorithm.

The paper does not explicitly introduce new benchmark datasets.

Key Insights and Ideas

  • The "chicken-and-egg" problem of timbre modeling: The paper highlights the difficulty in polyphonic transcription where knowing note shapes is needed to find notes, and knowing notes is needed to find their shapes.
  • The power of sparsity: The concept of sparse coding is presented as a powerful tool for music transcription, exploiting the fact that only a few notes are active at any given time. This leads to more efficient representations and potentially better separation.
  • ICA's limitations and extensions: While standard ICA is useful for simple mixtures, the paper points out its limitations with Gaussian sources and the permutation problem. It then discusses how extensions like considering convolutive mixing and incorporating prior knowledge (like source amplitude variations) are necessary for realistic scenarios.
  • Timbre's role in transcription: The paper discusses how timbre (instrument sound quality) is crucial for polyphonic transcription but can be challenging to model, especially when it changes over time. Monophonic transcription, however, is less affected by timbre variations.
  • Audio coding as a significant application: Beyond producing musical scores, the paper suggests that high-quality music transcription could enable new forms of audio coding and compression by describing music as a collection of "objects" (instruments and scores) rather than raw audio data.

Key Results

  • Monophonic Transcription: The autocorrelation-based system, combined with onset detection, was shown to produce a "good 'piano-roll' representation" for monophonic music, successfully transcribing a jazz solo trumpet piece, though some nuance of the original rendition was lost due to pitch bending.
  • Polyphonic Transcription (Blackboard System): The blackboard system showed promising results for polyphonic music transcription, with an extract of Liszt's Etude being transcribed very closely to the original. For a richer piece like Mussorgski's Promenade, the transcription was generally good, but note durations, especially for higher frequencies, were shorter than the original, indicating challenges with decaying notes.
  • Multiple-Cause Model: This model successfully learned to separate spectra from different notes of the same instrument and also from different instruments playing the same note, requiring around 800 training patterns. It also showed promise in separating real audio signals, though some instruments (flute and clarinet) were not perfectly separated.
  • Sparse Coding: The algorithm learned 55 basic spectra for Bach's Partita, with 49 identified as note spectra. A simple re-synthesis produced a "passable rendition" of the original piece, with some missing notes and timing jitters.
  • Source Separation (Convolutive Mixing): The proposed algorithm using a likelihood ratio jump test was successful in separating speech sources with non-trivial echo, outperforming the Smaragdis algorithm which failed in such conditions.

Limitations and Future Work

  • Timbre Modeling: The current sparse coding method assumes relatively constant timbre, which is a limitation for instruments with significant timbre changes over time (e.g., harpsichord). Future work needs to address this.
  • Non-harmonic Instruments: The polyphonic analysis methods are not suitable for instruments without a simple harmonic structure, such as percussion instruments.
  • Piano Harmonics: Even seemingly simple instruments like the piano can have complex harmonic structures with phase drifts, posing challenges for transcription.
  • Permutation Problem: While addressed in some contexts, the permutation problem in ICA (determining the correct order of sources) remains a consideration, especially for linked problems.
  • Temporal Structure: The paper acknowledges that it has largely focused on frequency content and has "largely ignored the temporal structure" of music, which is crucial for aspects like rhythm and beat structure.
  • Automatic Re-ordering of Note Shapes: In sparse coding, manual re-ordering of note shapes was required. Future work aims to develop automatic re-ordering methods.
  • Quality of Analysis: The multiple-cause model, while promising, "does not yet give us a quality of analysis that we would need for accurate transcription."

The paper concludes that musical audio analysis and separation is a "very interesting and fruitful area for future research," with potential for improved audio coding, compression, and tools for musicologists.

Summary updated

BibTeX
@article{plumbley2002automatic,
  author = {Mark D. Plumbley and Samer A. Abdallah and Juan Pablo Bello and Mike E. Davies and Giuliano Monti and Mark B. Sandler},
  title = {{Automatic Music Transcription and Audio Source Separation}},
  journal = {Cybern. Syst.},
  year = {2002},
  doi = {10.1080/01969720290040777},
}
Figure