TL;DR
This paper presents a subjective evaluation of state-of-the-art musical audio source separation algorithms, focusing on the challenging task of isolating the singing voice. The study reveals significant differences in perceived quality between algorithms and highlights that even top-performing systems struggle with song-specific variations, suggesting a need for more robust and generalized separation techniques.
What Problem Does This Paper Address?
Separating individual instruments and vocals from a mixed audio track (source separation) is a crucial task with applications ranging from music production and remixing to karaoke and even musicological analysis. While objective metrics like SDR (Source-to-Distortion Ratio) and PEASS (Perceptual Evaluation methods for Audio Source Separation) are widely used to evaluate algorithm performance, they don't always perfectly align with human perception of quality. This paper addresses the gap between objective performance and subjective listener experience, specifically for the difficult task of extracting the singing voice from complex musical mixtures. The authors aim to identify which algorithms are perceived by humans as best performing this task and to understand how well current objective metrics predict these subjective judgments.
Methods
The researchers conducted a listening experiment involving 34 participants who were presented with 13 professionally mixed songs. For each song, they were given six different versions of the isolated singing voice, each produced by a different state-of-the-art source separation algorithm. Participants were asked to select the version they perceived as most similar to the original vocal. The algorithms were pre-selected based on their high objective performance scores from the SiSEC 2018 MUS (Music) task. The study then analyzed the listener choices to determine which algorithms were consistently preferred and compared these subjective results against objective metrics (SDR and PEASS) to assess their predictive power.
Datasets
The primary dataset used in this study is MUSDB18. This is a publicly available dataset comprising 150 professionally mixed songs, each containing four stereo sources: bass, drums, vocals, and other instruments. For the SiSEC 2018 MUS task, MUSDB18 was split into 100 songs for training and 50 for testing. The authors specifically focused on the vocal stems from the test portion of MUSDB18 for their listening test.
Key Insights and Ideas
- Subjective quality matters: Even with high objective scores, algorithms can perform very differently in terms of perceived quality by human listeners.
- Song-dependency is a major challenge: State-of-the-art systems show significant variability in their performance across different songs, indicating that current algorithms may not generalize well to diverse musical content.
- Objective metrics have limitations: While SDR showed some correlation with subjective preferences, PEASS (OPS) had weaker and even negative correlations for some songs, suggesting that current objective measures may not fully capture the nuances of human perception for singing voice separation.
- Some algorithms consistently preferred: Despite song-dependency, certain algorithms (TAU1, TAK2, UHL3) were generally favored by listeners over others.
Key Results
- Out of the six evaluated algorithms, TAU1 was most frequently selected as the best separation for seven out of the 13 songs.
- For the Sunshine Garcia Band track, there was a notable divergence in listener preferences, with JY3 being chosen significantly more often than expected, and other algorithms performing poorly for this specific song.
- The study found that for seven songs, the most frequently selected algorithm also had the highest SDR score.
- However, the correlation between subjective results and OPS (from PEASS) was weaker, with some moderate negative correlations observed, indicating that OPS did not reliably predict listener preferences.
Limitations and Future Work
The authors acknowledge that the song-dependent performance of the algorithms is a significant limitation, suggesting that further research is needed to improve the generalization capabilities of source separation systems. They also highlight the need for refinement of objective measures, particularly PEASS, to better align with human perception of singing voice separation quality. Future work could involve investigating the specific characteristics of songs that lead to performance variations and developing algorithms that are more robust to these differences. Additionally, exploring new objective metrics that more accurately capture perceptual aspects of separation would be beneficial.
Summary updated