Preliminary guidelines for subjective evalutation of audio source separation algorithms

Emmanuel Vincent, Maria G. Jafari, Mark D. Plumbley2006

TL;DR

This paper addresses the challenge of reliably evaluating audio source separation algorithms by proposing standardized guidelines for subjective listening tests. It adapts the established MUSHRA test procedure to this specific domain, offering a more robust and reproducible method than ad-hoc approaches, and provides a free MATLAB tool to facilitate its implementation.

What Problem Does This Paper Address?

Audio source separation aims to disentangle individual sound sources (like voices or instruments) from a mixed audio signal. While objective metrics like SDR (Signal-to-Distortion Ratio) exist, they often fail to capture the nuances of human auditory perception, especially when multiple types of distortions are present. Subjective listening tests are crucial for assessing perceived quality and intelligibility, but previous attempts in source separation have lacked standardization, leading to inconsistent and unreproducible results. This paper tackles this gap by proposing a standardized methodology for subjective evaluation, drawing from best practices in related audio fields.

Methods

The core of the paper's contribution lies in adapting the MUSHRA (Multiple Stimulus with Hidden Reference and Anchors) standard for subjective listening tests. MUSHRA is a well-established ITU recommendation for evaluating audio quality. The authors propose the following adaptations for audio source separation:

  • Reference Signals: Mixtures are generated by either recording physical sources sequentially or convolving single-channel sources with synthetic filters. The spatial image of each source within the mixture is used as the reference.
  • Anchors: Three types of anchors are introduced to provide absolute quality ratings for specific distortions:
  • Interference Anchors: Created by adding a scaled version of other source images to the target source.
  • Noise Anchors: Created by adding scaled white noise to the target source.
  • Artifacts Anchors: Generated by randomly cancelling half of the time-frequency points in the target source's Short-Time Fourier Transform (STFT), using specific windowing and overlap parameters.
  • MUSHRAM MATLAB Interface: A free, GPL-licensed MATLAB graphical interface is provided to streamline the execution of MUSHRA tests, making the proposed methodology accessible to researchers.

Datasets

The paper does not introduce new datasets. Instead, it describes the generation of synthetic mixtures for evaluation purposes. For their example application, they used speech mixtures with varying reverberation times (RT) and signal-to-noise ratios (SNR). The specific audio files used in their experiments are made available on their webpage.

Key Insights and Ideas

  • Subjective tests are essential: Objective metrics, while useful, cannot fully replace human perception for evaluating audio quality, especially in complex scenarios like source separation.
  • Standardization is key: The lack of standardized procedures has hindered reliable comparison of source separation algorithms. Adapting existing standards like MUSHRA offers a path forward.
  • Anchors matter: Well-defined anchors are critical for ensuring the accuracy and reproducibility of subjective tests. The proposed anchors for interference, noise, and artifacts aim to provide a consistent baseline for comparison.
  • Perceptual complexity: Signals with multiple types of distortion can be perceived as more annoying than those with a single distortion, even if the loudness of the distortion is similar. This highlights the need for nuanced evaluation methods.
  • Accessibility: Providing a free software tool (MUSHRAM) significantly lowers the barrier to entry for researchers wanting to conduct standardized subjective evaluations.

Key Results

The paper presents results from an experiment evaluating three source separation algorithms (FDICA, DUET, and ASB) on speech mixtures under different acoustic conditions (varying RT and SNR).

  • Algorithm Performance: The Adaptive Stereo Basis (ASB) method performed best in clean conditions but worst in noisy, reverberant conditions.
  • Anchor Ratings: Interestingly, the anchors were rated higher than the actual estimated sources in some cases, suggesting the difficulty of the tested mixtures and the potential for anchors to represent less complex distortions.
  • Significance: The results demonstrate that subjective ratings can reveal performance differences between algorithms that might not be apparent with objective metrics alone.

Limitations and Future Work

  • Preliminary Guidelines: The authors emphasize that these guidelines are preliminary and intended as a starting point for collaborative standard definition.
  • Defining Distortions: Further work is needed to define multiple subjective ratings that correspond to different types of distortions and to understand how these combine to form overall quality ratings. The current categories (interference, noise, artifacts) might be insufficient to capture the full perceptual complexity of separated sources.
  • Generalizability: More experiments are needed to investigate the performance of different algorithms across a wider range of conditions and to refine the proposed guidelines.
  • Music Data: While the paper focuses on speech, adapting these guidelines for music source separation would be a valuable future direction.

Summary updated

Abstract

Evaluating audio source separation algorithms means rating the quality or intelligibility of separated source signals. While objective criteria fail to account for all auditory phenomena so far, precise subjective ratings can be obtained by means of listening tests. In practice, the accuracy and the reproducibility of these tests depend on several design issues. In this paper, we discuss some of these issues based on ongoing research in other areas of audio signal processing. We propose preliminary guidelines to evaluate the basic audio quality of separated sources and provide an example of their application using a free Matlab graphical interface.
BibTeX
@misc{vincent2006preliminary,
  author = {Emmanuel Vincent and Maria G. Jafari and Mark D. Plumbley},
  title = {{Preliminary guidelines for subjective evalutation of audio source separation algorithms}},
  year = {2006},
}
Figure