Adaptive filtering for music/voice separation exploiting the repeating musical structure

Antoine Liutkus, Zafar Rafii, Roland Badeau, Bryan Pardo, Gaël Richard

TL;DR

This paper introduces an enhanced adaptive filtering technique for separating singing voices from musical accompaniments in full audio tracks. By generalizing the REPET algorithm to handle time-varying repeating musical structures and employing soft time-frequency masking, the method achieves comparable or better performance than existing approaches while remaining computationally efficient.

What Problem Does This Paper Address?

Separating individual musical instruments or vocal tracks from a mixed audio recording is a long-standing challenge in audio signal processing. Specifically, isolating a singing voice from its musical background (music/voice separation) has numerous practical applications, including automatic karaoke, melody transcription for music databases, and improving speech recognition in noisy environments. While previous methods like REPET (REpeating Pattern Extraction Technique) showed promise by identifying and extracting repeating musical patterns, they were limited to short, stable excerpts and often produced artifacts due to binary masking. This paper addresses the need for a method that can effectively process entire musical tracks, accommodating variations in the background (e.g., verse vs. chorus) and producing cleaner separations.

Methods

The core of this work is an extension of the REPET algorithm. The original REPET relied on a fixed repeating period for the background music. This new approach generalizes REPET by introducing time-varying period estimation for the background. This is achieved by first computing a "beat spectrogram," which captures rhythmic variations across different frequency channels. Then, a dynamic programming approach is used to estimate the local spectral period, $T_0(t)$, for each time frame.

Instead of using a binary time-frequency (TF) mask (where a TF bin is either entirely assigned to the voice or the music), the paper proposes a soft TF masking strategy. This is based on comparing the observed TF bin energy with the estimated background energy. A Gaussian radial basis function is used to map this comparison to a soft weight between 0 and 1, allowing for proportional energy assignment to each source. This soft masking is intended to reduce the "noise artifacts" often associated with binary masking. The separation itself is performed using an adaptive Wiener-like filter, which applies the derived TF mask to the mixture.

Datasets

The authors evaluated their method on a newly created dataset of 14 complete musical tracks. These tracks were derived from recordings by The Beach Boys, where original accompaniments and vocals were available. The dataset was augmented by creating three versions of each track with different voice-to-music ratios (-6 dB, 0 dB, and 6 dB) to simulate various mixing scenarios. This dataset differs from previous work by allowing the evaluation of methods on full-length songs with inherent structural variations, rather than just short, stable excerpts.

Key Insights and Ideas

  • Generalizing REPET for Real-World Music: The key insight is that musical backgrounds are rarely perfectly periodic across an entire song. By allowing the repeating period to vary over time, the algorithm can handle common musical structures like verses and choruses, making it applicable to full tracks.
  • Soft Masking for Cleaner Separation: Moving from binary to soft TF masking is crucial. This allows for a more nuanced separation, reducing the harsh artifacts that can arise when TF bins are strictly classified, leading to perceptually better results.
  • Blind and Efficient Separation: The method is "blind" in that it doesn't require prior knowledge of the music or voice segments. Its reliance on self-similarity within the audio makes it computationally efficient and automatable.

Key Results

The proposed adaptive filtering method, when using soft TF masking and a 100 Hz high-pass filter on the voice estimate, achieved performance comparable to or better than the Multipass Median Filtering-based Separation (MMFS) method, a competitive music/voice separation technique. Specifically:

  • For music estimates, the proposed method showed higher Source-to-Interference Ratio (SIR) and similar Source-to-Distortion Ratio (SDR) compared to MMFS, suggesting better removal of vocal "interferences" at the cost of potentially introducing some separation artifacts (lower Source-to-Artifact Ratio, SAR).
  • For voice estimates, the method performed better when the voice was louder (higher voice-to-music ratio), indicating its effectiveness in extracting foreground vocals from the background.
  • The average computation time was approximately 1.18 seconds per second of audio on a standard PC, demonstrating its efficiency for processing complete recordings.

Limitations and Future Work

The authors acknowledge that their current probabilistic model for the background is not fully rigorous and is a subject of ongoing work. Future directions include:

  • Developing more thorough probabilistic models for the background.
  • Enabling the simultaneous separation of multiple repeating patterns within a single track.
  • Incorporating dynamic features into the separation process to leverage musicological knowledge.
  • Allowing users to manually specify track structures to facilitate separation.

Summary updated

Abstract

International audience
BibTeX
@inproceedings{liutkus2012adaptive,
  author = {Antoine Liutkus and Zafar Rafii and Roland Badeau and Bryan Pardo and Gaël Richard},
  title = {{Adaptive filtering for music/voice separation exploiting the repeating musical structure}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2012},
  doi = {10.1109/icassp.2012.6287815},
}
Figure