TL;DR
This paper challenges the conventional wisdom that speech separation requires "one-to-many" (SIMO) architectures. By comparing standard designs against mixed and purely iterative "one-to-one" (SISO) models, the authors demonstrate that incorporating SISO layers improves performance—especially in low-overlap scenarios—without increasing the total number of parameters.
What Problem Does This Paper Address?
In the world of deep learning-based speech separation, most models follow a Single-Input Multi-Output (SIMO) paradigm: you feed in one mixture and get out $C$ separated tracks simultaneously. While some researchers have added Single-Input Single-Output (SISO) modules for "post-enhancement," these are often treated as optional add-ons that simply increase the model's size.
The authors of this paper ask a fundamental architectural question: Why do SISO modules help, and is the SIMO layer even necessary? They seek to understand if the standard "one-to-many" mapping is the most efficient way to use a fixed budget of neural network layers, or if a different arrangement of SIMO and SISO components could yield better results.
Methods
The researchers used DPRNN-TasNet as their backbone architecture, keeping the total number of blocks ($M=6$) constant across all experiments to ensure a fair comparison of parameter efficiency. They tested three primary configurations:
- SIMO-only: The standard approach where all blocks contribute to estimating $C$ masks at once.
- Mixed SIMO-SISO: The model is split. First, $K$ blocks perform a "pre-separation" (SIMO), then $M-K$ blocks act as a shared "post-enhancement" module (SISO) that processes each estimated track individually.
- SISO-only: A purely iterative approach. The model uses an encoder-decoder structure to extract one speaker at a time. In each step, it uses the residual signal (what's left of the mixture) as a "bias" to find the next speaker.
Datasets
The models were evaluated on a simulated noisy reverberant two-speaker dataset.
- Speech Sources: 100-hour subset of LibriSpeech.
- Noise: 100 Non-speech Corpus.
- Conditions: Overlap ratios were uniformly sampled from 0% to 100%, with added reverberation (T60 between 0.1 and 0.5 seconds) and random signal-to-noise ratios.
- Scale: 20,000 training, 5,000 validation, and 3,000 test utterances.
Key Insights and Ideas
- The Power of Post-Enhancement: SISO layers are exceptionally good at "cleaning up" signals. The authors found that assigning roughly 70% of the model's depth to the SISO module (2 SIMO blocks followed by 4 SISO blocks) provided the best overall performance.
- SISO Excels in Low Overlap: The mixed SIMO-SISO design showed the most significant gains in scenarios where speakers don't overlap much. In these "single-speaker regions," standard SIMO models struggle because their output layers are forced to reconstruct both salient speech and silence using the same parameters. Deeper SISO layers alleviate this constraint.
- SIMO is Optional: Perhaps the most surprising finding is that SISO-only models can match or beat standard SIMO models. By iteratively extracting speakers, a purely "one-to-one" network can perform blind source separation effectively, challenging the idea that we need a "one-to-many" layer at the end of the network.
Key Results
The performance was measured using Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) in decibels (dB):
- Standard SIMO-only: 9.0 dB average.
- Best Mixed SIMO-SISO (2 SIMO / 4 SISO): 9.5 dB average. This configuration showed a nearly 1 dB improvement in low-overlap cases (<25% overlap).
- Best SISO-only: 9.3 dB average. Even without a dedicated SIMO layer, this iterative approach outperformed the standard baseline.
Limitations and Future Work
- Fixed Speaker Count: The current study assumes the number of speakers is known in advance. While the iterative SISO approach is naturally suited for an unknown number of speakers, this wasn't the primary focus of the evaluation.
- Feature Fusion Mystery: The authors noted that feeding both the intermediate features and the original mixture into the SISO module consistently improved results, but the underlying theoretical reason for this "feature fusion" success remains a topic for future research.
- Bias Information: In the SISO-only model, the "bias" is currently just the residual signal. Future work could explore using more sophisticated biases, such as speaker identities or content-related features, to further refine the iterative extraction process.
Summary updated