Abstract
Modules in all existing speech separation networks can be categorized into single-input-multi-output (SIMO) modules and single-input-single-output (SISO) modules. SIMO modules generate more outputs than input, and SISO modules keep the numbers of input and output the same. While the majority of separation models only contain SIMO architectures, it has also been shown that certain two-stage separation systems integrated with a post-enhancement SISO module can improve the separation quality. Why performance improvements can be achieved by incorporating the SISO modules? Are SIMO modules always necessary? In this paper, we empirically examine those questions by designing models with varying configurations in the SIMO and SISO modules. We show that comparing with the standard SIMO-only design, a mixed SIMO-SISO design with a same model size is able to improve the separation performance especially under low-overlap conditions. We further validate the necessity of SIMO modules and show that SISO-only models are still able to perform separation without sacrificing the performance. The observations allow us to rethink the model design paradigm and present different views on how the separation is performed.
Introduction
In the field of deep learning-based speech separation, the standard approach relies on a Single-Input Multi-Output (SIMO) paradigm. In this setup, a network maps a single mixture input to $C$ separated tracks simultaneously. While some researchers have integrated Single-Input Single-Output (SISO) modules for "post-enhancement," these are often viewed as optional additions that simply increase model size. This paper investigates a fundamental architectural question: Is the standard "one-to-many" mapping the most efficient use of a fixed budget of neural network layers, or could a different arrangement of SIMO and SISO components yield better results?
Architectural Configurations
To compare these paradigms fairly, the researchers used DPRNN-TasNet as a backbone, keeping the total number of blocks ($M=6$) constant across all experiments. They evaluated three primary configurations:
- SIMO-only: The standard approach where all blocks contribute to estimating $C$ masks at once.
- Mixed SIMO-SISO: The model is split. First, $K$ blocks perform a "pre-separation" (SIMO), then $M-K$ blocks act as a shared "post-enhancement" module (SISO) that processes each estimated track individually.
- SISO-only: A purely iterative approach. The model uses an encoder-decoder structure to extract one speaker at a time, using the residual signal as a "bias" to find the next speaker.
Experimental Setup
The models were evaluated on a simulated noisy reverberant two-speaker dataset based on a 100-hour subset of LibriSpeech. Overlap ratios were uniformly sampled from 0% to 100%, with added reverberation (T60 between 0.1 and 0.5 seconds) and random signal-to-noise ratios. The dataset consisted of 20,000 training, 5,000 validation, and 3,000 test utterances.
Results and Discussion
Performance was measured using the Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) in decibels (dB).
The Power of Post-Enhancement
The study found that SISO layers are exceptionally effective at signal refinement. Assigning roughly 70% of the model's depth to the SISO module (specifically, 2 SIMO blocks followed by 4 SISO blocks) provided the best overall performance, reaching an average of 9.5 dB compared to the 9.0 dB baseline.
SISO Excels in Low Overlap
The mixed SIMO-SISO design showed the most significant gains in scenarios with low speaker overlap (<25%). In these regions, standard SIMO models struggle because their output layers are forced to reconstruct both salient speech and silence using the same parameters. Deeper SISO layers alleviate this constraint by focusing on individual signal cleaning.
SIMO is Optional
Perhaps the most surprising finding is that SISO-only models can match or beat standard SIMO models. By iteratively extracting speakers, a purely "one-to-one" network achieved a 9.3 dB average, challenging the assumption that a "one-to-many" layer is essential for separation.
Conclusion
This research demonstrates that a mixed SIMO-SISO design improves separation performance, particularly in low-overlap conditions, without increasing the total number of parameters. Furthermore, the success of SISO-only models suggests a paradigm shift toward iterative, one-to-one extraction methods. Future work will explore the theoretical reasons behind the success of "feature fusion" and investigate more sophisticated bias mechanisms for iterative extraction.
Summary updated