Introduction
Automatic Music Transcription (AMT)—the process of converting raw audio into symbolic notation like MIDI—is a foundational task in Music Information Retrieval (MIR). However, real-world music is complex, often featuring multiple instruments with overlapping frequencies. Most existing AMT systems are either "instrument-blind" or restricted to specific instruments like piano or drums.
Jointist is a multi-task deep learning framework designed to address these limitations. It simultaneously performs instrument recognition, multi-instrument transcription, and source separation. By jointly learning these related tasks, the model produces instrument-specific piano rolls and demonstrates that these symbolic representations can significantly enhance downstream music analysis tasks such as chord recognition and beat tracking.
Methodology
Jointist consists of three interconnected modules that work in synergy to process audio signals:
- Instrument Recognition ($f_{IR}$): A Transformer-based tagger that identifies which of the 39 supported instrument classes are present in the audio.
- Transcription ($f_T$): An "onsets-and-frames" module that generates piano rolls. It is conditioned by the instrument recognition module using FiLM (Feature-wise Linear Modulation), allowing a single architecture to adapt its behavior based on the target instrument.
- Source Separation ($f_{MSS}$): This module isolates the audio of specific instruments. It utilizes both the instrument tag and the predicted piano roll from the transcription module as "hints" to improve separation quality.
The framework employs a joint training objective where the total loss is the sum of losses from all three tasks. A key finding was the necessity of pretraining the transcription module; without it, early-stage "noisy" transcriptions can lead the source separator into poor local minima.
Experimental Setup
The model was trained and evaluated using the Slakh2100 dataset, a massive collection of synthesized music featuring 2,100 tracks and 167 synthesizer plugins. These plugins were mapped into 39 distinct instrument classes (e.g., Acoustic Guitar, Saxophone) to balance realism with manageability. For downstream evaluation, standard benchmarks like Isophonics and RWC-POP were used for chord and key estimation.
Key Results and Discussion
Transcription and Source Separation
Jointist achieved an instrument-wise F1 score of 24.8, vastly outperforming previous instrument-aware baselines like Omnizart (F1: 1.9). In source separation, the joint training approach improved the Source-to-Distortion Ratio (SDR). By using transcription results as a condition, the SDR for individual instruments rose from 1.52 dB (standalone) to 2.01 dB.
Downstream Task Enhancement
One of the most compelling insights from this work is that a piano roll acts as a "denoised" version of music, stripping away timbre and recording noise to provide a clean harmonic and rhythmic map. The authors demonstrated that adding these symbolic piano rolls to standard spectrograms (a "hybrid" approach) consistently improved performance in downstream tasks.
As shown in the results, the hybrid model outperformed audio-only inputs across downbeat detection, chord recognition, and key estimation. For instance, chord recognition accuracy improved from 0.798 to 0.812 when using the hybrid input.
Limitations and Future Work
Despite its successes, Jointist faces challenges with class imbalance, struggling with rare instruments like the harp. Additionally, while symbolic rolls help with harmonic tasks, they perform worse than raw audio for genre or mood classification because they lack timbre information. Future work may involve replacing Recurrent Neural Network components with pure Transformers to better capture long-term musical dependencies.
Summary updated