TL;DR
This paper introduces "Cerberus," a novel deep learning architecture that can simultaneously separate musical instruments from a mixture and transcribe each instrument's notes into a piano roll format. This unified approach leads to improved performance on both tasks compared to independent models, demonstrating the synergistic benefits of joint learning.
What Problem Does This Paper Address?
Humans can effortlessly listen to a musical piece, pick out individual instruments, and even mentally transcribe what they're hearing. However, achieving this dual capability in artificial systems has been a significant challenge. Existing audio source separation systems focus solely on isolating instruments, while music transcription systems typically work on isolated or simpler monophonic sources. When dealing with complex mixtures of multiple polyphonic and percussive instruments, both tasks become considerably harder. This paper tackles the problem of building a single system that can perform both separation and transcription simultaneously, leveraging a shared musical representation to improve performance on both fronts.
Methods
The core innovation is the "Cerberus" architecture, which builds upon the Chimera network for source separation. Cerberus extends Chimera by adding a third "head" specifically for transcription. The architecture consists of:
- Shared Body: A stack of bidirectional Long Short-Term Memory (BLSTM) layers that process the input magnitude spectrogram, learning a shared representation.
- Three Heads:
- Deep Clustering Head: Learns an embedding space to group time-frequency bins belonging to the same source.
- Mask Inference Head: Generates masks to separate the audio sources.
- Transcription Head: Produces a piano roll transcription (note onsets and offsets) for each instrument.
The system is trained using a weighted combination of three loss functions: deep clustering loss ($\mathcal{L}_{DC}$), mask inference loss ($\mathcal{L}_{MI}$), and transcription loss ($\mathcal{L}_{TR}$). This multitask learning approach allows the network to learn a representation that benefits both separation and transcription.
Datasets
- Synthesized Data: The primary training dataset is Slakh2100, a collection of synthesized musical mixtures with accompanying isolated sources and MIDI transcriptions. The authors created four sets of mixtures with varying instrument combinations (piano+guitar, piano+guitar+bass, piano+guitar+bass+drums, piano+guitar+bass+drums+strings).
- Real-World Data: For evaluation on real instruments, the authors generated a dataset by combining segments from:
- MAPS: A dataset of classical piano performances.
- GuitarSet: A dataset of guitar performances. These real-world mixtures were designed to be "incoherent" (instruments from different songs), making them more challenging and dissimilar to the training data.
Key Insights and Ideas
- Synergy between Separation and Transcription: The paper strongly suggests that learning separation and transcription jointly leads to better performance on both tasks than training them independently. The shared representation learned by Cerberus is more effective for both.
- Complementarity of Tasks: The findings indicate that separation and transcription are complementary. For instance, combining deep clustering (a separation-related task) with transcription loss resulted in a significant improvement in separation performance.
- Generalization Benefits: The Cerberus model, trained with all three loss functions, demonstrated better generalization to unseen, real-world, and incoherent musical mixtures compared to single-task networks.
- Scalability Challenges: While Cerberus shows promise, performance predictably degrades as the number of simultaneous instruments in a mixture increases.
Key Results
- Improved Performance: The Cerberus model, trained with all three loss functions, outperformed single-task networks and the Chimera network on both separation (measured by SDR) and transcription (measured by precision, recall, and F1-score) on synthesized data.
- Competitive Transcription: The transcription performance on isolated piano was competitive with state-of-the-art models for isolated piano transcription.
- Real-World Evaluation: On the generated real-world dataset, Cerberus showed a significant drop in performance, highlighting the domain gap, but still outperformed single-task networks.
- Instrument-Specific Performance: While overall performance degrades with more instruments, bass and drums could still be reasonably separated and transcribed even in complex five-instrument mixtures.
Limitations and Future Work
- Domain Gap: The significant performance drop when evaluating on real-world data compared to synthesized data indicates a need for more robust models or better real-world datasets.
- Incoherent Mixtures: The current system performs best on coherent mixtures (instruments from the same song) and struggles with incoherent ones.
- Scalability: Performance degrades with an increasing number of instruments.
- Future Directions: The authors suggest exploring more advanced network architectures, incorporating dedicated losses for note onsets and velocities, and training on real multi-instrument recordings with aligned score data.
Summary updated