TL;DR
This paper introduces GuitarDuets, a novel dataset of real and synthesized classical guitar duets, to address the challenging problem of monotimbral music source separation (MSS). They adapt the state-of-the-art Demucs model and propose a joint transcription-separation framework, demonstrating that combining real and synthesized data improves separation performance, especially when guided by note-level information.
What Problem Does This Paper Address?
The field of Music Source Separation (MSS) has largely focused on separating distinct instrument families (e.g., vocals, drums, bass) in a mix – known as multi-timbral separation. However, separating instruments with very similar timbral characteristics, such as two classical guitars playing together (monotimbral separation), remains a significant challenge. This is because existing models often rely on timbral differences to distinguish sources, and a lack of suitable datasets hinders research in this specific area. The authors aim to bridge this gap by creating a dedicated dataset and evaluating separation techniques for this challenging monotimbral scenario.
Methods
The core of the proposed methodology involves adapting the Hybrid Transformer Demucs architecture for monotimbral separation. Demucs is a powerful MSS model that operates in both the time and spectrogram domains using U-Nets. The authors modify it to output two stereo signals, one for each guitar.
Key methodological contributions include:
- Dataset Creation: The GuitarDuets dataset, comprising approximately three hours of audio, is introduced. It includes both real recordings of classical guitar duets and synthesized recordings generated from MIDI scores using virtual instruments. The synthesized portion is particularly valuable as it comes with note-level annotations.
- Score-Informed Separation: They explore conditioning the Demucs model on note-level information. This is achieved by concatenating binary vectors representing the presence or absence of specific MIDI notes within temporal frames into the model's intermediate feature maps. This conditioning can happen in either the temporal or spectral branches, or both.
- Joint Transcription and Separation: To address scenarios where ground-truth note labels are unavailable, they develop a framework that jointly trains a transcription network (using the Residual Shuffle-Exchange Network, RSE) and the separation network. The transcription network generates piano-roll representations, which are then used to inform the separation process. A permutation-invariant loss is employed to handle the ambiguity of assigning sources to outputs.
Datasets
- GuitarDuets: The primary dataset introduced in this paper.
- Real Recordings (GuitarDuets(R)): Approximately 58.6 minutes of real classical guitar duet recordings.
- Synthesized Recordings (GuitarDuets(S)): Approximately 106 minutes of synthesized classical guitar duet recordings, generated from MIDI scores and featuring note-level annotations.
- GuitarSet: Used for training the transcription network and for cross-dataset evaluation. This dataset consists of acoustic guitar recordings with detailed annotations.
- musdb18, MoisesDB, URMP, SLAKH, EnsembleSet: Mentioned for comparison with existing MSS datasets, highlighting the novelty of GuitarDuets in its focus on monotimbral duets.
Key Insights and Ideas
- Timbral Similarity is the Bottleneck: The paper strongly suggests that the primary difficulty in separating classical guitar duets lies in their similar timbres, making it harder for models and even evaluation metrics to distinguish them.
- Data Fusion is Beneficial: Combining both real and synthesized data for training leads to improved separation performance on an unseen test set compared to using either subset alone. This highlights the complementary strengths of each data type.
- Ground Truth is King, Predictions are Helpful (but less so): While having perfect, ground-truth note-level annotations significantly boosts separation performance, using predicted note estimates from a joint transcription model provides only a marginal improvement. This suggests that the accuracy of the transcription is crucial for its effectiveness.
- Metrics Need Careful Interpretation: The paper points out that standard MSS metrics like SDR and SI-SDR might behave differently and potentially overestimate performance when applied to sources with very similar timbres, as seen in the guitar duet scenario.
Key Results
- The GuitarDuets dataset provides a valuable resource for monotimbral MSS research, offering a significant amount of audio and note-level annotations for synthesized duets.
- Training the Demucs model on the combined GuitarDuets dataset (real + synthesized) yielded the highest SDR values on the testing set.
- The score-informed separation approach, particularly when conditioning on ground-truth note labels in both temporal and spectral branches, showed improved separation quality, especially for the second guitar, which is typically harder to separate.
- The joint transcription-separation framework showed some improvement for real guitar duets but performed better when the transcription network was trained on data it hadn't seen during separation training.
- The paper discusses the behavior of SDR and SI-SDR, noting that these metrics tend to report higher scores for monotimbral mixtures compared to multitimbral ones, suggesting a potential overestimation of performance due to timbral similarity.
Limitations and Future Work
- Limited Duration of Real Data: The real recordings subset of GuitarDuets is relatively short (approx. 58.6 minutes), which can be a limitation for training deep learning models.
- Cross-Microphone Bleeding: The real recordings suffer from some cross-microphone sound bleeding, which was partially addressed by a specialized test set.
- Marginal Improvement from Predicted Notes: The predicted note estimates from the joint transcription model offered only a marginal improvement in separation, suggesting room for improvement in the transcription accuracy or its integration.
- Future Work: The authors plan to:
- Extend the duration of both real and synthesized recordings.
- Provide note-level annotations for the real subset of GuitarDuets.
- Explore more sophisticated methods for integrating predicted guitar transcripts into the separation model.
- Conduct extensive listening tests to complement objective metric evaluations.
Summary updated