TL;DR
This paper introduces a novel "Multi-Layer Non-negative Matrix Factor Deconvolution" (ML-NMFD) method for separating harmonic and percussive sounds in single-channel audio. By leveraging pre-trained time-frequency dictionaries of real drum sounds for the percussive component, their approach achieves superior drum extraction and overall source separation compared to existing methods, while also handling non-stationary harmonic sounds more effectively.
What Problem Does This Paper Address?
Extracting individual instruments from a mixed audio signal, especially when dealing with complex sounds like drums, is a challenging task in audio signal processing. Traditional Non-negative Matrix Factorization (NMF) struggles with non-stationary sounds like drum hits, often decomposing them into multiple, physically meaningless components. This makes it difficult to accurately identify and isolate drum sounds, particularly in music with multiple instruments. The paper aims to improve Harmonic/Percussive Source Separation (HPSS) by developing a method that can better represent and extract these non-stationary percussive elements, specifically drums, while simultaneously handling harmonic components.
Methods
The core of the proposed method is a "Multi-Layer NMFD" (ML-NMFD). This approach extends the Non-negative Matrix Factor Deconvolution (NMFD) technique. While standard NMF decomposes a signal into a dictionary of patterns and their activations, NMFD uses time-frequency (TF) atoms that can vary over time, making them better suited for non-stationary sounds.
The ML-NMFD specifically addresses drum extraction by:
- Fixed Drum Dictionaries: Instead of learning drum patterns from scratch, the method uses pre-trained, fixed time-frequency dictionaries derived from real drum sounds (e.g., bass drum, snare, hi-hat). These dictionaries are built from the ENST-Drum database.
- Multi-Layer Decomposition: The ML-NMFD decomposes the audio signal into two layers:
- Percussive Layer: This layer utilizes the NMFD with the fixed drum dictionaries to explicitly model and extract the drum sounds.
- Harmonic Layer: This layer employs a sparse orthogonal component model (similar to Projected NMF) to capture the harmonic content of the music.
- Optimization: The model is optimized using a Kullback-Leibler (KL) divergence cost function, with multiplicative update rules derived for the percussive and harmonic components. Crucially, since the drum dictionaries are fixed, the optimization process is sped up as these templates do not need to be updated.
- Signal Reconstruction: After separation, Wiener filtering is used to reconstruct the percussive and harmonic signals, preserving their phase information.
Datasets
- ENST-Drum database: Used for constructing the fixed time-frequency dictionaries for drum sounds.
- Medley-dB database: A collection of 89 multitrack audio songs containing percussive/drum sounds, used for evaluating the proposed method's performance in Harmonic/Percussive Source Separation (HPSS).
Key Insights and Ideas
- Leveraging Prior Knowledge for Drums: The key insight is that by using pre-trained, real-world drum sound dictionaries, the NMFD can achieve a more physically meaningful decomposition of percussive elements. This contrasts with methods that learn dictionaries from the mixed signal, which can lead to less interpretable results.
- Multi-Layered Approach for Specialized Tasks: The paper demonstrates the effectiveness of a multi-layer approach where different decomposition techniques are applied to different signal components (percussive vs. harmonic). This allows for specialized handling of each type of sound.
- Improved Robustness to Non-Stationary Harmonics: The method shows robustness not only for drums but also for non-stationary harmonic sources, which are often poorly handled by existing techniques.
- Physical Interpretability: The use of real drum sound templates leads to a decomposition that is more interpretable, as the activation of a template directly corresponds to a specific drum event.
Key Results
- Superior Separation Performance: The ML-NMFD consistently outperformed other state-of-the-art methods (Constrained NMF, Median Filtering, Structured Projective NMF) on the Medley-dB dataset in terms of Signal-to-Distortion Ratio (SDR) and Signal-to-Interference Ratio (SIR) for both percussive and harmonic source separation.
- Effective Drum Extraction: The method showed a significant improvement in extracting drum sounds, achieving higher SIR scores.
- Better Vocal Handling: When vocals were present, the ML-NMFD effectively separated them into the harmonic part, unlike other methods that often split them between harmonic and percussive components, leading to a lower SIR for the overall decomposition.
- Computational Cost: While more computationally intensive than some methods (e.g., Median Filtering), the ML-NMFD's performance gains justify its use.
Limitations and Future Work
- Computational Intensity: The authors acknowledge that their ML-NMFD algorithm is the most computationally intensive among the compared methods.
- Future Work: The authors suggest several promising directions:
- Using the ML-NMFD for drum transcription.
- Exploring drum re-synthesis and drum remixing applications.
- Further investigation into the method's potential as a pre-processing step for other audio analysis algorithms.
Summary updated