TL;DR
This paper introduces TAGBOX, a novel unsupervised method for separating musical sources (like vocals, drums, or guitar) from a mixture. It cleverly repurposes large, pre-trained generative music models (like Jukebox) and music taggers, steering them through gradient ascent in the latent space to isolate desired instruments without any retraining. This approach significantly expands the range of separable sources compared to existing methods, highlighting the untapped potential of large pre-trained models for audio tasks.
What Problem Does This Paper Address?
The field of music source separation, a crucial task for music analysis and manipulation, faces a significant bottleneck: the scarcity of labeled data. Most existing supervised methods require extensive datasets of isolated instrument tracks, which are expensive and time-consuming to create. This data limitation restricts current systems to separating only a few common instrument classes (e.g., vocals, bass, drums). While unsupervised methods exist, they often rely on hand-designed algorithms or specific musical priors that limit their flexibility. This paper aims to overcome these limitations by leveraging the vast knowledge embedded in large, pre-trained generative music models, which have been trained on massive amounts of unlabeled audio, to achieve flexible and broad-spectrum source separation.
Methods
The core of TAGBOX lies in its innovative use of pre-trained models without any fine-tuning. The process works as follows:
- Input Mixture Encoding: An input audio mixture is fed into the encoder of a pre-trained generative model (specifically, Jukebox's VQ-VAE). This produces a latent embedding representing the audio.
- Latent Space Steering: Instead of directly generating audio, the system performs gradient ascent on this latent embedding. The goal is to modify the embedding such that when the corresponding audio is decoded, it matches a predefined set of target instrument tags (e.g., "guitar").
- Mask Generation: The audio decoded from the steered embedding is used to create a mask. This mask is applied to the original input mixture's spectrogram. The idea is to remove the parts of the mixture that correspond to the desired source, effectively isolating it.
- Tagging and Loss Calculation: The masked audio (representing the removed portion) is then passed through a pre-trained music tagger. The cross-entropy loss between the tagger's output and the target tags is calculated.
- Gradient Ascent: This loss signal is used to update the latent embedding via gradient ascent, iteratively refining the embedding to better match the target tags.
- Source Extraction: The final separated source is obtained by subtracting the audio estimated to be removed (from the gradient ascent process) from the original input mixture.
Crucially, the weights of both the generative model (Jukebox VQ-VAE) and the music tagger remain frozen throughout this process. The system only manipulates the latent representation.
Datasets
- Training: The paper leverages pre-trained models that were trained on large, unlabeled datasets. Specifically, Jukebox was trained on 1.2 million songs.
- Evaluation:
- MUSDB18: A standard dataset for music source separation, containing 150 songs with isolated sources for vocals, bass, drums, and "other."
- Slakh2100: A dataset of synthesized musical mixtures with isolated sources for bass, drums, guitar, piano, and strings. The authors selected 50 songs from the test set for evaluation.
Key Insights and Ideas
- Repurposing Pre-trained Models: The most significant insight is that large, pre-trained generative models and taggers, originally designed for different tasks, can be effectively repurposed for source separation without any task-specific retraining.
- Latent Space Steering as a Mechanism: The paper demonstrates that gradient ascent in the latent space of a generative model can be a powerful tool for controlling audio generation towards specific semantic targets (in this case, instrument tags).
- Unsupervised Flexibility: TAGBOX offers unprecedented flexibility in source separation. By simply changing the target tags provided to the music tagger, the system can attempt to separate a wide variety of instruments, limited only by the tagger's vocabulary. This contrasts sharply with supervised methods that require retraining for each new source.
- Leveraging Implicit Priors: The success of TAGBOX suggests that the vast amounts of unlabeled data used to train models like Jukebox embed rich implicit priors about music structure and instrument characteristics, which can be exploited for downstream tasks like separation.
Key Results
- Broad Source Separation: TAGBOX demonstrates the ability to separate a wider range of sources than many existing purpose-built supervised systems. Table 1 shows it can separate vocals, bass, drums, guitar, piano, and strings, outperforming some unsupervised methods and being competitive with supervised ones on certain metrics.
- Competitive SDRi: While not always outperforming state-of-the-art supervised systems, TAGBOX achieves considerable Source-to-Distortion Ratio Improvement (SDRi) across various source types. For instance, on MUSDB18, it achieves SDRi values in the range of 7.1-7.8 dB for vocals, bass, and drums, and on Slakh2100, it achieves SDRi values around 9.3 dB for guitar and 10.5 dB for strings.
- Tagger Influence: The choice of pre-trained tagger architecture and dataset impacts separation quality. The paper found that FCN architectures generally performed better than HarmonicCNN, and MTG-Jamendo taggers often yielded better results than MTAT taggers.
- Perceptual vs. Quantitative Trade-off: While quantitative metrics like SDRi are promising, the authors note that the perceptual quality of the separated audio can sometimes be lacking, with apparent artifacts. However, they also suggest that techniques like multi-scale spectral loss and longer gradient ascent steps can improve perceptual results.
Limitations and Future Work
- Perceptual Quality: The authors acknowledge that the perceptual quality of the separated sources can be a limitation, with artifacts sometimes present.
- Computational Cost: Performing gradient ascent for each separation instance can be computationally expensive, especially for long audio segments or when many steps are required.
- Tagger Limitations: The system's ability to separate sources is inherently limited by the available tags in the music tagger. Overlapping or ambiguous tags (e.g., "voice" vs. "vocals") can pose challenges.
- Future Work:
- Exploring techniques to improve perceptual quality, such as multi-scale spectral loss and longer gradient ascent.
- Investigating other pre-trained generative models and taggers.
- Developing more efficient methods for gradient ascent or alternative steering mechanisms.
- Exploring the use of non-instrument tags (e.g., genre, mood) for creative audio manipulation tasks.
- Further leveraging the shared priors in large pre-trained models to address data scarcity in MIR tasks.
Summary updated