A Two-Stage Phase-Aware Approach for Monaural Multi-Talker Speech Separation

Lu Yin, Junfeng Li, Yonghong Yan, Masato AkagiIEICE Trans. Information and Systems 2020 · OpenReview

Abstract

Simultaneous utterances pose a significant challenge for both hearing-impaired individuals and automatic speech recognition (ASR) systems. While deep neural networks (DNNs) have substantially improved speech separation performance, most existing methods focus solely on estimating spectral magnitude while relying on the original mixture phase for signal reconstruction. This reliance on the "mixture phase" creates a performance bottleneck. This study proposes a two-stage phase-aware approach that integrally recovers both magnitude and phase. By utilizing the Multiple Input Spectrogram Inversion (MISI) algorithm with an Ideal Amplitude Mask (IAM) for phase recovery, and a novel Phase-Recovered Mask (PRM) for magnitude estimation, the proposed framework significantly minimizes signal distortion. Experimental results on the WSJ0-2mix and TIMIT datasets demonstrate the effectiveness and generalization capabilities of the approach.

Introduction

The "cocktail party problem"—isolating individual voices from a single-channel (monaural) recording—remains a central challenge in audio processing. Traditional deep learning approaches typically estimate the spectral magnitude (the loudness at various frequencies) but reuse the phase (the timing of the waves) from the noisy mixture. Because the mixture phase is a complex combination of all active speakers, using it for reconstruction leads to audible artifacts and limited separation quality.

Estimating phase is notoriously difficult due to its seemingly unstructured nature. This research aims to break through this "phase ceiling" by introducing a framework that treats phase recovery as a primary objective rather than an afterthought.

Proposed Two-Stage Framework

The core of the proposed system is a cascaded architecture that separates the problem into two distinct but linked stages: phase recovery and magnitude estimation.

Overview of the two-stage phase-aware speech separation framework, integrating phase recovery and magnitude estimation.

Stage 1: Phase Recovery

In the first stage, a Bi-directional Long Short-Term Memory (BiLSTM-I) network estimates an Ideal Amplitude Mask (IAM). This mask serves as the input for the Multiple Input Spectrogram Inversion (MISI) algorithm. MISI is an iterative process that alternates between the time and frequency domains to find a phase that is mathematically consistent with the estimated magnitude.

A key finding of this study is that the IAM is the optimal starting point for MISI-based phase recovery. While previous literature often suggested the Phase-Sensitive Mask (PSM), the authors demonstrate that the IAM provides a cleaner target, resulting in lower phase distortion during the iterative refinement process.

Stage 2: Magnitude Estimation

Since phase recovery is rarely perfect, the second stage is designed to compensate for residual errors. The authors introduce the Phase-Recovered Mask (PRM), a novel mask trained to minimize the Mean Square Error (MSE) between the separated signal and the clean ground truth, specifically accounting for the timing errors left by Stage 1.

The PRM acts as a mathematical bridge: if phase recovery is perfect, the PRM converges to the IAM; if phase is unrecovered (mixture phase), it behaves like the PSM. This flexibility allows the network to "mop up" distortions and fine-tune the volume levels to match the reconstructed phase waves.

Architectural Variants

The study evaluates two neural network configurations for the second stage to test flexibility and performance:

  1. BiLSTM-II: Employs parallel, independent networks to estimate the PRM for each speaker.
  2. BiLSTM-III: Utilizes a single, unified network to handle all speakers simultaneously. This architecture incorporates a Phase Alignment step to ensure that the estimated magnitudes are correctly mapped to their corresponding recovered phases.
The BiLSTM-III architecture, which jointly estimates magnitudes for multiple speakers and incorporates a phase alignment step.

Experimental Evaluation

The models were trained on the WSJ0-2mix dataset (20,000 utterances) and tested for generalization on the TIMIT dataset.

Key Results

  • Superior Performance: The two-stage approach achieved a Signal-to-Distortion Ratio (SDR) of 11.81 dB on WSJ0-2mix, outperforming standard baselines like uPIT-ST (10.0 dB) and Chimera++ (11.5 dB).
  • Phase Impact: Incorporating the phase recovery stage (using IAM with 6 iterations) provided a significant 1.59 dB boost in SDR compared to using the noisy mixture phase.
  • Generalization: The model maintained high performance on the TIMIT dataset, indicating that it learns the underlying physics of speech rather than simply memorizing training patterns.

Conclusion

This research demonstrates that integrating phase recovery into the speech separation pipeline is essential for high-fidelity results. By cascading magnitude estimation after phase recovery, the network can adapt to the specific timing of reconstructed signals. Future work will focus on reducing the computational complexity of the iterative MISI algorithm to enable real-time applications and exploring end-to-end optimization by unfolding iterations directly into the neural architecture.

Summary updated

Abstract

The simultaneous utterances impact the ability of both the hearing-impaired persons and automatic speech recognition systems. Recently, deep neural networks have dramatically improved the speech separation performance. However, most previous works only estimate the speech magnitude and use the mixture phase for speech reconstruction. The use of the mixture phase has become a critical limitation for separation performance. This study proposes a two-stage phase-aware approach for multi-talker speech separation, which integrally recovers the magnitude as well as the phase. For the phase recovery, Multiple Input Spectrogram Inversion (MISI) algorithm is utilized due to its effectiveness and simplicity. The study implements the MISI algorithm based on the mask and gives that the ideal amplitude mask (IAM) is the optimal mask for the mask-based MISI phase recovery, which brings less phase distortion. To compensate for the error of phase recovery and minimize the signal distortion, an advanced mask is proposed for the magnitude estimation. The IAM and the proposed mask are estimated at different stages to recover the phase and the magnitude, respectively. Two frameworks of neural network are evaluated for the magnitude estimation on the second stage, demonstrating the effectiveness and flexibility of the proposed approach. The experimental results demonstrate that the proposed approach significantly minimizes the distortions of the separated speech.
BibTeX
@article{yin2020two,
  author = {Lu Yin and Junfeng Li and Yonghong Yan and Masato Akagi},
  title = {{A Two-Stage Phase-Aware Approach for Monaural Multi-Talker Speech Separation}},
  journal = {IEICE Trans. Information and Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=xkCVXVvU7A},
}
Figure