Two-Step Sound Source Separation: Training On Learned Latent Targets

Efthymios Tzinis, Shrikant Venkataramani, Zhepei Wang, Y. Cem Sübakan, Paris Smaragdis

TL;DR

This paper introduces a novel two-step training strategy for sound source separation. Instead of training an end-to-end model, they first learn an optimal latent representation and then train a separation module within that learned space. This approach consistently outperforms traditional end-to-end methods, achieving better separation quality by leveraging a more effective latent representation.

What Problem Does This Paper Address?

Single-channel audio source separation, the task of extracting individual sound sources from a mixed audio signal, is a long-standing challenge in audio processing. While deep learning has significantly advanced this field, many current methods rely on end-to-end training of an encoder, a separation module, and a decoder. However, these joint training approaches can sometimes lead to suboptimal latent representations, limiting the ultimate separation performance. The paper aims to address this by proposing a method to learn a more effective latent space before training the separation module, thereby improving overall separation quality.

Methods

The core of the paper's contribution is a two-step training procedure:

  1. Step 1: Learning the Latent Targets: This step focuses on training an encoder and a decoder to learn a latent representation. The goal is to find a transformation into a latent space where separation using masking would be optimal. This is achieved by training the encoder-decoder pair using a permutation-invariant Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) loss function applied to the reconstructed time-domain signals. The key idea is to create a latent space that "facilitates" source separation.
  2. Step 2: Training the Separation Module: Once the encoder and decoder are fixed, a separation module is trained. This module operates on the latent representation of the mixture and is trained to estimate the latent representations of the individual clean sources. Crucially, the loss function used here is the permutation-invariant SI-SDR applied directly in the latent space. The authors prove that optimizing SI-SDR in this learned latent space provides a lower bound for SI-SDR in the time domain, justifying this approach.

Datasets

The paper evaluates its method on three distinct sound separation tasks:

  • Speech Separation: Uses the Wall Street Journal (WSJO) corpus, specifically the publicly available WSJ0-2mix dataset.
  • Non-Speech Separation: Utilizes the ESC50 dataset, which contains environmental sound clips. To increase diversity, an augmented mixture generation process is employed.
  • Mixed Separation: Combines sources from both WSJO and ESC50 datasets, allowing for mixtures of speech and non-speech sounds.

Key Insights and Ideas

  • Pre-learning an optimal latent space is crucial: The paper strongly suggests that the quality of the latent representation significantly impacts separation performance. Learning this space first, rather than jointly with the separation module, leads to better results.
  • Latent space SI-SDR as a proxy for time-domain SI-SDR: The theoretical proof that latent space SI-SDR lower-bounds time-domain SI-SDR is a key insight, providing a solid foundation for the proposed method.
  • Learned latent representations are more structured and sparse: The experiments show that the latent representations learned via the two-step approach are sparser and exhibit a more organized, spectrogram-like structure compared to those from joint end-to-end training. This suggests a more efficient encoding of the audio.
  • Generalizability: The proposed methodology is presented as general and applicable to a wide range of neural network-based end-to-end separation systems.

Key Results

  • Consistent performance improvement: The two-step approach consistently achieved higher SI-SDR improvement (SI-SDRi) across all tested tasks (speech, non-speech, and mixed separation) compared to end-to-end training with the same network architecture.
  • Up to 0.7dB improvement: The absolute SI-SDR improvement over the end-to-end baseline reached up to 0.7dB for speech, non-speech, and mixed separation tasks.
  • Higher oracle mask performance: The latent space oracle mask obtained with the two-step approach showed a significantly higher upper bound for separation performance compared to ideal masks in the STFT domain.
  • Sparser and more structured latent representations: Qualitative analysis (Figure 2) demonstrated that the learned latent representations were sparser (l1 norm ~10x smaller) and more structured, with distinct patterns for different sound types.

Limitations and Future Work

  • Limitations: The paper focuses on a specific type of decoder (1D transposed convolutional layer) for its theoretical proofs. While the methodology is claimed to be general, the theoretical guarantees are tied to this specific architecture. The performance gains, while consistent, are in the range of 0.5-0.7 dB, which might be considered incremental by some.
  • Future Work: The authors suggest that this approach can be easily adapted for use with other mask-based systems beyond the TDCN architectures they tested. Further exploration into different latent space properties and their impact on separation could be a direction. The generalizability to more complex mixtures (more than two sources) or different audio domains could also be investigated.

Summary updated

Abstract

In this paper, we propose a two-step training procedure for source separation via a deep neural network. In the first step we learn a transform (and it's inverse) to a latent space where masking-based separation performance using oracles is optimal. For the second step, we train a separation module that operates on the previously learned space. In order to do so, we also make use of a scale-invariant signal to distortion ratio (SI-SDR) loss function that works in the latent space, and we prove that it lower-bounds the SI-SDR in the time domain. We run various sound separation experiments that show how this approach can obtain better performance as compared to systems that learn the transform and the separation module jointly. The proposed methodology is general enough to be applicable to a large class of neural network end-to-end separation systems.
BibTeX
@inproceedings{tzinis2020two,
  author = {Efthymios Tzinis and Shrikant Venkataramani and Zhepei Wang and Y. Cem Sübakan and Paris Smaragdis},
  title = {{Two-Step Sound Source Separation: Training On Learned Latent Targets}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=VnzlwDov60X},
}

Loading related content...

Figure