Jointist: Joint Learning for Multi-instrument Transcription and Its Applications

Kin Wai Cheuk, Keunwoo Choi, Qiuqiang Kong, Bochen Li, Minz Won, Amy Hung, Ju-Chiang Wang, Dorien HerremansarXiv 2022 · DOI · arXiv

Introduction

Automatic Music Transcription (AMT)—the process of converting raw audio into symbolic notation like MIDI—is a foundational task in Music Information Retrieval (MIR). However, real-world music is complex, often featuring multiple instruments with overlapping frequencies. Most existing AMT systems are either "instrument-blind" or restricted to specific instruments like piano or drums.

Jointist is a multi-task deep learning framework designed to address these limitations. It simultaneously performs instrument recognition, multi-instrument transcription, and source separation. By jointly learning these related tasks, the model produces instrument-specific piano rolls and demonstrates that these symbolic representations can significantly enhance downstream music analysis tasks such as chord recognition and beat tracking.

Methodology

Jointist consists of three interconnected modules that work in synergy to process audio signals:

  1. Instrument Recognition ($f_{IR}$): A Transformer-based tagger that identifies which of the 39 supported instrument classes are present in the audio.
  2. Transcription ($f_T$): An "onsets-and-frames" module that generates piano rolls. It is conditioned by the instrument recognition module using FiLM (Feature-wise Linear Modulation), allowing a single architecture to adapt its behavior based on the target instrument.
  3. Source Separation ($f_{MSS}$): This module isolates the audio of specific instruments. It utilizes both the instrument tag and the predicted piano roll from the transcription module as "hints" to improve separation quality.
Overview of the Jointist architecture, illustrating the flow from audio input to instrument recognition, transcription, and source separation.

The framework employs a joint training objective where the total loss is the sum of losses from all three tasks. A key finding was the necessity of pretraining the transcription module; without it, early-stage "noisy" transcriptions can lead the source separator into poor local minima.

Experimental Setup

The model was trained and evaluated using the Slakh2100 dataset, a massive collection of synthesized music featuring 2,100 tracks and 167 synthesizer plugins. These plugins were mapped into 39 distinct instrument classes (e.g., Acoustic Guitar, Saxophone) to balance realism with manageability. For downstream evaluation, standard benchmarks like Isophonics and RWC-POP were used for chord and key estimation.

Key Results and Discussion

Transcription and Source Separation

Jointist achieved an instrument-wise F1 score of 24.8, vastly outperforming previous instrument-aware baselines like Omnizart (F1: 1.9). In source separation, the joint training approach improved the Source-to-Distortion Ratio (SDR). By using transcription results as a condition, the SDR for individual instruments rose from 1.52 dB (standalone) to 2.01 dB.

Downstream Task Enhancement

One of the most compelling insights from this work is that a piano roll acts as a "denoised" version of music, stripping away timbre and recording noise to provide a clean harmonic and rhythmic map. The authors demonstrated that adding these symbolic piano rolls to standard spectrograms (a "hybrid" approach) consistently improved performance in downstream tasks.

Performance comparison of audio-only vs. hybrid (audio + transcription) inputs for downstream music analysis tasks.

As shown in the results, the hybrid model outperformed audio-only inputs across downbeat detection, chord recognition, and key estimation. For instance, chord recognition accuracy improved from 0.798 to 0.812 when using the hybrid input.

Limitations and Future Work

Despite its successes, Jointist faces challenges with class imbalance, struggling with rare instruments like the harp. Additionally, while symbolic rolls help with harmonic tasks, they perform worse than raw audio for genre or mood classification because they lack timbre information. Future work may involve replacing Recurrent Neural Network components with pure Transformers to better capture long-term musical dependencies.

Summary updated

Abstract

In this paper, we introduce Jointist, an instrument-aware multi-instrument framework that is capable of transcribing, recognizing, and separating multiple musical instruments from an audio clip. Jointist consists of the instrument recognition module that conditions the other modules: the transcription module that outputs instrument-specific piano rolls, and the source separation module that utilizes instrument information and transcription results. The instrument conditioning is designed for an explicit multi-instrument functionality while the connection between the transcription and source separation modules is for better transcription performance. Our challenging problem formulation makes the model highly useful in the real world given that modern popular music typically consists of multiple instruments. However, its novelty necessitates a new perspective on how to evaluate such a model. During the experiment, we assess the model from various aspects, providing a new evaluation perspective for multi-instrument transcription. We also argue that transcription models can be utilized as a preprocessing module for other music analysis tasks. In the experiment on several downstream tasks, the symbolic representation provided by our transcription model turned out to be helpful to spectrograms in solving downbeat detection, chord recognition, and key estimation.
BibTeX
@misc{cheuk2022jointist,
  author = {Kin Wai Cheuk and Keunwoo Choi and Qiuqiang Kong and Bochen Li and Minz Won and Amy Hung and Ju-Chiang Wang and Dorien Herremans},
  title = {{Jointist: Joint Learning for Multi-instrument Transcription and Its Applications}},
  year = {2022},
  eprint = {2206.10805},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.10805},
}
Figure