On Synthesis for Supervised Monaural Speech Separation in Time Domain

Jingjing Chen, Qirong Mao, Dong Liu2020 · DOI

TL;DR

This paper introduces a "synthesis method" for time-domain monaural speech separation that can reduce artifacts in separated speech, especially in noisy conditions. By directly learning source estimations in an encoded domain, this approach can synthesize lost audio information, leading to improved separation performance without additional computational cost.

What Problem Does This Paper Address?

Monaural speech separation, the task of isolating a target speech signal from a single-channel audio mixture, is a challenging problem with significant real-world applications like improving automatic speech recognition in noisy environments. While recent time-domain deep learning approaches have shown great promise, they often struggle with broadband noise artifacts in the separated sources, particularly when the original mixture is noisy. Existing methods, especially those relying on masking in the encoded domain, are limited because they cannot synthesize new audio information to recover what might have been lost due to noise or overlapping speech. This paper aims to address this limitation by proposing a novel synthesis approach.

Methods

The core contribution is the "synthesis method," which differs from traditional "masking methods" in time-domain speech separation.

  • Masking Method: In this approach, an encoder transforms the input mixture into an encoded domain. A separation network then learns to estimate masks for each source. These masks are multiplied with the encoded representation to obtain estimated source signals, which are then decoded back into the time domain. The limitation here is that masks operate linearly and cannot synthesize lost information.
  • Synthesis Method: This new approach also uses an encoder-decoder framework. However, instead of learning masks, the separation network directly learns to estimate the source signals in the encoded domain. This allows the network to "synthesize" audio information that might have been lost in the original mixture due to noise or overlaps. The authors emphasize that this synthesis refers to reconstructing audio features, not text-to-speech synthesis.

The synthesis method is designed to be "plug-and-play," meaning it can be seamlessly integrated into existing time-domain speech separation architectures like Conv-TasNet and DPRNN without requiring modifications to their core encoder-decoder structures or incurring extra computational overhead.

Datasets

The authors created two datasets based on the Librispeech dataset [23] and a noise corpus [24]:

  • LS-2mix: A common monaural dataset for two-speaker speech separation. Mixtures were generated with varying Signal-to-Noise Ratios (SNRs) between 0 dB and 5 dB.
  • LS-2mixNoise: A noisy monaural dataset where the two-speaker mixtures from LS-2mix were further corrupted by random noise signals sampled from a noise corpus. SNRs for the noise were randomly chosen between 0 dB and 5 dB.

Key Insights and Ideas

  • Synthesizing Lost Information is Crucial for Noisy Separation: The most significant insight is that the ability to synthesize audio, rather than just mask existing features, is key to effectively separating speech in the presence of noise. Masking methods are inherently limited in their ability to recover information lost due to noise.
  • "Plug-and-Play" Effectiveness: The synthesis method is not a standalone system but a module that enhances existing architectures. Its seamless integration into Conv-TasNet and DPRNN without extra computation is a major practical advantage.
  • Noise Intensity Amplifies Synthesis Benefits: The paper observed that the performance gains of the synthesis method over the masking method become more pronounced as the noise level in the mixture increases. This highlights its particular utility in challenging, noisy scenarios.
  • Generalization Beyond Training SNRs: The synthesis method demonstrated superior performance even on mixtures with SNRs not encountered during training, suggesting good generalization capabilities.

Key Results

The synthesis method consistently outperformed the masking method, especially on noisy datasets:

  • On LS-2mixNoise (using Conv-TasNet): The synthesis method achieved a relative improvement of 0.97 dB (10.1%) SDR, along with gains in SIR, SAR, and STOI, without additional computation.
  • On LS-2mixNoise (using DPRNN): The synthesis method showed improvements of 0.97 dB SDR, 6.16 dB SIR, 0.92 dB SAR, and 0.02 STOI over the masking method.
  • Across Different SNRs (using DPRNN): The synthesis method consistently outperformed the masking method, with performance gains increasing as SNRs decreased (i.e., as noise increased). For example, at 0 dB SNR, the SDR improvement was 1.05 dB.
  • Qualitative Results: Visualizations of magnitude spectrograms showed that the synthesis method produced cleaner and more complete source estimations, particularly in regions where the masking method introduced artifacts due to noise.

Limitations and Future Work

  • Limitations: While the paper focuses on monaural separation, the authors do not explicitly discuss limitations related to the complexity of the underlying models (Conv-TasNet, DPRNN) or potential issues with very low SNRs beyond what was tested. The "synthesis" is within the encoded domain, not a generative model in the broader sense.
  • Future Work: The authors suggest that the synthesis method's ability to generate lost sounds could be beneficial for other speech processing tasks. The paper's success in noisy conditions also opens avenues for further exploration in more complex acoustic environments.

Summary updated

BibTeX
@misc{chen2020synthesis,
  author = {Jingjing Chen and Qirong Mao and Dong Liu},
  title = {{On Synthesis for Supervised Monaural Speech Separation in Time Domain}},
  year = {2020},
  doi = {10.21437/interspeech.2020-1150},
}
Figure