Abstract
A natural question arising in Music Source Separation (MSS) is whether long range contextual information is useful, or whether local acoustic features are sufficient. In other fields, attention based Transformers have shown their ability to integrate information over long sequences. In this work, we introduce Hybrid Transformer Demucs (HT Demucs), an hybrid temporal/spectral bi-U-Net based on Hybrid Demucs, where the innermost layers are replaced by a cross-domain Transformer Encoder, using self-attention within one domain, and cross-attention across domains. While it performs poorly when trained only on MUSDB, we show that it outperforms Hybrid Demucs (trained on the same data) by 0.45 dB of SDR when using 800 extra training songs. Using sparse attention kernels to extend its receptive field, and per source fine-tuning, we achieve state-of-the-art results on MUSDB with extra training data, with 9.20 dB of SDR.
Introduction
Music Source Separation (MSS)—the task of isolating vocals, drums, bass, and other instruments from a complete mix—has traditionally relied on local acoustic features or Recurrent Neural Networks (RNNs). While Transformers have revolutionized fields like NLP and Computer Vision by capturing long-range dependencies, their application in MSS has been hindered by data scarcity and the extreme computational complexity of processing high-fidelity audio signals. This paper introduces HT Demucs, an architecture that integrates a cross-domain Transformer into a dual-domain U-Net to leverage long-range context for superior separation quality.
Architecture and Methods
The HT Demucs architecture is built upon the Hybrid Demucs framework, which uses two parallel U-Nets to process audio in both the temporal (raw waveform) and spectral (spectrogram) domains.
Cross-Domain Transformer Encoder
The researchers replaced the innermost layers of the U-Nets with a specialized Transformer Encoder. This component is designed to find patterns within a single domain via self-attention and share information between domains via cross-attention.
Sparse Attention for Long Sequences
Processing several seconds of 44.1kHz audio leads to massive memory consumption. To mitigate this, the authors utilized Locally Sensitive Hashing (LSH) sparse attention. This allows the model to handle sequences up to 12.2 seconds long by attending only to the most relevant parts of the signal, reducing memory usage by approximately 90%.
Experimental Setup
Data Requirements
Transformers are notoriously "data-hungry." The authors found that training HT Demucs solely on the standard MUSDB18 dataset (87 training songs) resulted in worse performance than the baseline. To unlock the Transformer's potential, they curated an additional 800 high-quality training songs, which provided the necessary scale for the model to learn complex dependencies.
Training and Fine-Tuning
The model was trained using extensive data augmentation, including remixing stems from different songs and repitching. Following the initial training of a multi-target model, the researchers performed per-source fine-tuning—creating dedicated models for individual instruments—to achieve peak performance.
Results
HT Demucs set a new state-of-the-art on the MUSDB18 benchmark with a Signal-to-Distortion Ratio (SDR) of 9.20 dB, surpassing previous models like Band-Split RNN.
Key insights from the evaluation include:
- Context Matters: Increasing the audio "chunk" size from 3.4s to 7.8s during training yielded a +0.53 dB SDR boost.
- Efficiency: The model remains computationally efficient, with a Real-Time Factor (RTF) of 2.04 on a single CPU core.
- Transformer Gain: Compared to the original Hybrid Demucs trained on the same augmented dataset, the Transformer-based version provided a clear gain of 0.45 dB.
Conclusion
The success of HT Demucs highlights the importance of long-range context in music source separation. By combining the strengths of U-Nets for local feature extraction and Transformers for global dependency modeling, the architecture achieves superior isolation of musical components. The study also underscores the critical role of dataset scale and memory-efficient attention mechanisms in applying Transformers to high-dimensional audio data.
Summary updated