TL;DR
This paper introduces a novel adversarial semi-supervised learning framework for audio source separation, specifically applied to extracting singing voices. By leveraging large amounts of unlabelled music and solo instrument recordings alongside limited multi-track data, their method significantly improves separation performance compared to purely supervised approaches, making source separation more robust and scalable.
What Problem Does This Paper Address?
Separating individual audio sources (like vocals, drums, or instruments) from a mixed recording is a challenging task in music information retrieval. Current state-of-the-art methods primarily rely on supervised learning, training deep neural networks on multi-track datasets where the ground truth for each source is known. However, such multi-track datasets are scarce, leading to overfitting issues. While data augmentation is a common workaround, simply mixing random tracks can be unrealistic because real-world music sources are highly correlated, not independent. This correlation is precisely what makes separation difficult, and naive augmentation fails to capture it. The paper aims to overcome the limitations of data scarcity and unrealistic augmentation by developing a semi-supervised approach that can effectively learn from readily available unlabelled music and solo source recordings.
Methods
The core innovation is an adversarial semi-supervised learning framework. Instead of solely relying on a supervised loss that directly compares estimated sources to ground truth (as in standard supervised training), they introduce an unsupervised loss. This unsupervised loss drives the separator network to produce source estimates that are indistinguishable from real source signals, as judged by discriminator networks.
Here's a breakdown:
- Supervised Loss ($L_s$): This is the standard loss used in supervised source separation, minimizing the difference between the separator's output and the ground truth for paired multi-track data.
- Unsupervised Adversarial Loss ($L_u$): This is the novel component. For each source (e.g., singing voice), a dedicated discriminator network is trained to distinguish between real solo recordings of that source and the separator's estimates of that source from unlabelled mixtures. The separator is then trained to "fool" these discriminators, making its outputs more realistic. This is inspired by Generative Adversarial Networks (GANs).
- Wasserstein Distance: To improve training stability, the authors opt for the Wasserstein distance instead of the more common Jensen-Shannon or Kullback-Leibler divergences. This choice is motivated by the fact that Wasserstein distance provides a more stable gradient, preventing vanishing or exploding gradients during training, especially when dealing with limited sample sets.
- Additive Penalty ($L_{add}$): An additional loss term is included to ensure that the estimated sources, when summed together, approximate the original mixture. This is particularly important for magnitude spectrograms where phase information is not directly used, and it helps enforce the additive nature of audio signals.
The overall loss is a weighted sum of these components: $L = L_s + \alpha L_u + \beta L_{add}$.
Datasets
- Supervised Training Data ($D_m$): The training partition of the DSD100 (Deep Sound Separation Database) was used.
- Unlabelled Mixture Data ($D_u$): One-third of tracks from the iKala, MedleyDB, and CCMixter datasets were used.
- Source Datasets ($D_k$): For each source type (e.g., singing voice), one-third of tracks from iKala, MedleyDB, and CCMixter were used as real examples for the discriminators.
- Validation and Test Sets: These were constructed from the remaining thirds of iKala, MedleyDB, and CCMixter, plus 25 tracks from the DSD100 test partition.
The paper notes that using multi-track data for the unsupervised set (even though it's paired) was done to eliminate dataset bias and ensure that improvements stem from the semi-supervised learning itself. They also mention that larger datasets like DAMP could be used for unsupervised data if their bias is manageable.
Key Insights and Ideas
- Adversarial Priors for Source Realism: The core idea that a separator should produce outputs indistinguishable from real sources is powerful. This allows learning complex, realistic source characteristics without explicit modelling.
- Leveraging Unlabelled Data: The framework effectively taps into the vast amount of unlabelled music available, significantly expanding the training data beyond scarce multi-track recordings.
- Source-Specific Discriminators: Using a separate discriminator for each source (e.g., one for vocals, one for accompaniment) allows the model to learn distinct characteristics and priors for each.
- Wasserstein GAN for Stability: The adoption of Wasserstein distance for the adversarial loss is a crucial practical insight for achieving stable training in this complex domain.
- Semi-Supervised as a Surrogate: The unsupervised loss can act as a surrogate for the supervised loss, enabling learning from unlabelled data even when input-output pairings are unknown.
Key Results
- Performance Improvement: The proposed adversarial semi-supervised method consistently improved performance over the purely supervised baseline across various metrics (SDR, SIR, SAR) on test sets derived from MedleyDB, CCMixter, and iKala.
- Specific Gains: Notably, improvements were significant on the iKala dataset, demonstrating the ability to learn from unlabelled data without explicit pairings.
- Baseline Comparison: The baseline model performed better on the DSD100 subset, which the authors attribute to potential overfitting of the supervised model to its specific training data.
- Qualitative Analysis: Visualizations of the discriminator's gradients showed that it successfully learned to identify and penalize accompaniment artifacts in vocal estimates, encouraging the separator to produce cleaner vocal outputs.
Limitations and Future Work
- Data Scarcity Impact: The paper acknowledges that performance can decrease if the unlabelled data is too scarce or does not match the distribution of the test set.
- Future Directions:
- Applying the framework to multi-instrument separation, which involves more complex source priors.
- Training larger separator models that might not generalize well with purely supervised methods.
- Developing more sophisticated discriminator architectures to better distinguish real from generated samples and handle inherent source variability.
Summary updated