Rethinking the Separation Layers in Speech Separation Networks

Yi Luo, Zhuo Chen, Cong Han, Chenda Li, Tianyan Zhou, Nima Mesgarani

Abstract

Modules in all existing speech separation networks can be categorized into single-input-multi-output (SIMO) modules and single-input-single-output (SISO) modules. SIMO modules generate more outputs than input, and SISO modules keep the numbers of input and output the same. While the majority of separation models only contain SIMO architectures, it has also been shown that certain two-stage separation systems integrated with a post-enhancement SISO module can improve the separation quality. Why performance improvements can be achieved by incorporating the SISO modules? Are SIMO modules always necessary? In this paper, we empirically examine those questions by designing models with varying configurations in the SIMO and SISO modules. We show that comparing with the standard SIMO-only design, a mixed SIMO-SISO design with a same model size is able to improve the separation performance especially under low-overlap conditions. We further validate the necessity of SIMO modules and show that SISO-only models are still able to perform separation without sacrificing the performance. The observations allow us to rethink the model design paradigm and present different views on how the separation is performed.

Introduction

In the field of deep learning-based speech separation, the standard approach relies on a Single-Input Multi-Output (SIMO) paradigm. In this setup, a network maps a single mixture input to $C$ separated tracks simultaneously. While some researchers have integrated Single-Input Single-Output (SISO) modules for "post-enhancement," these are often viewed as optional additions that simply increase model size. This paper investigates a fundamental architectural question: Is the standard "one-to-many" mapping the most efficient use of a fixed budget of neural network layers, or could a different arrangement of SIMO and SISO components yield better results?

Architectural Configurations

To compare these paradigms fairly, the researchers used DPRNN-TasNet as a backbone, keeping the total number of blocks ($M=6$) constant across all experiments. They evaluated three primary configurations:

  1. SIMO-only: The standard approach where all blocks contribute to estimating $C$ masks at once.
  2. Mixed SIMO-SISO: The model is split. First, $K$ blocks perform a "pre-separation" (SIMO), then $M-K$ blocks act as a shared "post-enhancement" module (SISO) that processes each estimated track individually.
  3. SISO-only: A purely iterative approach. The model uses an encoder-decoder structure to extract one speaker at a time, using the residual signal as a "bias" to find the next speaker.
Architectural configurations: (A) standard SIMO-only, (B) Mixed SIMO-SISO with feature fusion, and (C) SISO-only iterative extraction.

Experimental Setup

The models were evaluated on a simulated noisy reverberant two-speaker dataset based on a 100-hour subset of LibriSpeech. Overlap ratios were uniformly sampled from 0% to 100%, with added reverberation (T60 between 0.1 and 0.5 seconds) and random signal-to-noise ratios. The dataset consisted of 20,000 training, 5,000 validation, and 3,000 test utterances.

Results and Discussion

Performance was measured using the Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) in decibels (dB).

The Power of Post-Enhancement

The study found that SISO layers are exceptionally effective at signal refinement. Assigning roughly 70% of the model's depth to the SISO module (specifically, 2 SIMO blocks followed by 4 SISO blocks) provided the best overall performance, reaching an average of 9.5 dB compared to the 9.0 dB baseline.

Performance comparison (SI-SDR) of mixed SIMO-SISO configurations across different overlap ratios.

SISO Excels in Low Overlap

The mixed SIMO-SISO design showed the most significant gains in scenarios with low speaker overlap (<25%). In these regions, standard SIMO models struggle because their output layers are forced to reconstruct both salient speech and silence using the same parameters. Deeper SISO layers alleviate this constraint by focusing on individual signal cleaning.

SIMO is Optional

Perhaps the most surprising finding is that SISO-only models can match or beat standard SIMO models. By iteratively extracting speakers, a purely "one-to-one" network achieved a 9.3 dB average, challenging the assumption that a "one-to-many" layer is essential for separation.

Conclusion

This research demonstrates that a mixed SIMO-SISO design improves separation performance, particularly in low-overlap conditions, without increasing the total number of parameters. Furthermore, the success of SISO-only models suggests a paradigm shift toward iterative, one-to-one extraction methods. Future work will explore the theoretical reasons behind the success of "feature fusion" and investigate more sophisticated bias mechanisms for iterative extraction.

Summary updated

Abstract

Modules in all existing speech separation networks can be categorized into single-input-multi-output (SIMO) modules and single-input-single-output (SISO) modules. SIMO modules generate more outputs than input, and SISO modules keep the numbers of input and output the same. While the majority of separation models only contain SIMO architectures, it has also been shown that certain two-stage separation systems integrated with a post-enhancement SISO module can improve the separation quality. Why performance improvements can be achieved by incorporating the SISO modules? Are SIMO modules always necessary? In this paper, we empirically examine those questions by designing models with varying configurations in the SIMO and SISO modules. We show that comparing with the standard SIMO-only design, a mixed SIMO-SISO design with a same model size is able to improve the separation performance especially under low-overlap conditions. We further validate the necessity of SIMO modules and show that SISO-only models are still able to perform separation without sacrificing the performance. The observations allow us to rethink the model design paradigm and present different views on how the separation is performed.
BibTeX
@inproceedings{luo2021rethinking,
  author = {Yi Luo and Zhuo Chen and Cong Han and Chenda Li and Tianyan Zhou and Nima Mesgarani},
  title = {{Rethinking the Separation Layers in Speech Separation Networks}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9414176},
}
Figure