SISEC 2018: state of the art in musical audio source separation - Subjective selection of the best algorithm

Dominic Ward, Russell Mason, Chungeun Kim, Fabian-Robert Stöter, Antoine Liutkus, Mark D. PlumbleySurrey 2018

TL;DR

This paper presents a subjective evaluation of state-of-the-art musical audio source separation algorithms, focusing on the challenging task of isolating the singing voice. The study reveals significant differences in perceived quality between algorithms and highlights that even top-performing systems struggle with song-specific variations, suggesting a need for more robust and generalized separation techniques.

What Problem Does This Paper Address?

Separating individual instruments and vocals from a mixed audio track (source separation) is a crucial task with applications ranging from music production and remixing to karaoke and even musicological analysis. While objective metrics like SDR (Source-to-Distortion Ratio) and PEASS (Perceptual Evaluation methods for Audio Source Separation) are widely used to evaluate algorithm performance, they don't always perfectly align with human perception of quality. This paper addresses the gap between objective performance and subjective listener experience, specifically for the difficult task of extracting the singing voice from complex musical mixtures. The authors aim to identify which algorithms are perceived by humans as best performing this task and to understand how well current objective metrics predict these subjective judgments.

Methods

The researchers conducted a listening experiment involving 34 participants who were presented with 13 professionally mixed songs. For each song, they were given six different versions of the isolated singing voice, each produced by a different state-of-the-art source separation algorithm. Participants were asked to select the version they perceived as most similar to the original vocal. The algorithms were pre-selected based on their high objective performance scores from the SiSEC 2018 MUS (Music) task. The study then analyzed the listener choices to determine which algorithms were consistently preferred and compared these subjective results against objective metrics (SDR and PEASS) to assess their predictive power.

Datasets

The primary dataset used in this study is MUSDB18. This is a publicly available dataset comprising 150 professionally mixed songs, each containing four stereo sources: bass, drums, vocals, and other instruments. For the SiSEC 2018 MUS task, MUSDB18 was split into 100 songs for training and 50 for testing. The authors specifically focused on the vocal stems from the test portion of MUSDB18 for their listening test.

Key Insights and Ideas

  • Subjective quality matters: Even with high objective scores, algorithms can perform very differently in terms of perceived quality by human listeners.
  • Song-dependency is a major challenge: State-of-the-art systems show significant variability in their performance across different songs, indicating that current algorithms may not generalize well to diverse musical content.
  • Objective metrics have limitations: While SDR showed some correlation with subjective preferences, PEASS (OPS) had weaker and even negative correlations for some songs, suggesting that current objective measures may not fully capture the nuances of human perception for singing voice separation.
  • Some algorithms consistently preferred: Despite song-dependency, certain algorithms (TAU1, TAK2, UHL3) were generally favored by listeners over others.

Key Results

  • Out of the six evaluated algorithms, TAU1 was most frequently selected as the best separation for seven out of the 13 songs.
  • For the Sunshine Garcia Band track, there was a notable divergence in listener preferences, with JY3 being chosen significantly more often than expected, and other algorithms performing poorly for this specific song.
  • The study found that for seven songs, the most frequently selected algorithm also had the highest SDR score.
  • However, the correlation between subjective results and OPS (from PEASS) was weaker, with some moderate negative correlations observed, indicating that OPS did not reliably predict listener preferences.

Limitations and Future Work

The authors acknowledge that the song-dependent performance of the algorithms is a significant limitation, suggesting that further research is needed to improve the generalization capabilities of source separation systems. They also highlight the need for refinement of objective measures, particularly PEASS, to better align with human perception of singing voice separation quality. Future work could involve investigating the specific characteristics of songs that lead to performance variations and developing algorithms that are more robust to these differences. Additionally, exploring new objective metrics that more accurately capture perceptual aspects of separation would be beneficial.

Summary updated

Abstract

The Signal Separation Evaluation Campaign (SiSEC) is a
\nlarge-scale regular event aimed at evaluating current progress
\nin source separation through a systematic and reproducible
\ncomparison of the participants’ algorithms, providing the
\nsource separation community with an invaluable glimpse of
\nrecent achievements and open challenges. This paper focuses
\non the music separation task from SiSEC 2018, which
\ncompares algorithms aimed at recovering instrument stems
\nfrom a stereo mix. In this context, we conducted a subjective
\nevaluation whereby 34 listeners picked which of six competing
\nalgorithms, with high objective performance scores,
\nbest separated the singing-voice stem from 13 professionally
\nmixed songs. The subjective results reveal strong differences
\nbetween the algorithms, and highlight the presence
\nof song-dependent performance for state-of-the-art systems.
\nCorrelations between the subjective results and the scores of
\ntwo popular performance metrics are also presented.
BibTeX
@inproceedings{ward2018sisec,
  author = {Dominic Ward and Russell Mason and Chungeun Kim and Fabian-Robert Stöter and Antoine Liutkus and Mark D. Plumbley},
  title = {{SISEC 2018: state of the art in musical audio source separation - Subjective selection of the best algorithm}},
  booktitle = {Surrey},
  year = {2018},
}
Figure