On The Compensation Between Magnitude and Phase in Speech Separation

Zhong-Qiu Wang, Gordon Wichern, Jonathan Le Roux

Abstract

Deep neural network (DNN) based end-to-end optimization in the complex time-frequency (T-F) domain or time domain has shown considerable potential in monaural speech separation. Many recent studies optimize loss functions defined solely in the time or complex domain, without including a loss on magnitude. Although such loss functions typically produce better scores if the evaluation metrics are objective time-domain metrics, they however produce worse scores on speech quality and intelligibility metrics and usually lead to worse speech recognition performance, compared with including a loss on magnitude. While this phenomenon has been experimentally observed by many studies, it is often not accurately explained and there lacks a thorough understanding on its fundamental cause. This paper provides a novel view from the perspective of the implicit compensation between estimated magnitude and phase. Analytical results based on monaural speech separation and robust automatic speech recognition (ASR) tasks in noisy-reverberant conditions support the validity of our view.

Introduction

Recent breakthroughs in speech separation have moved toward "end-to-end" models that operate directly on waveforms or complex-valued spectrograms. These models are typically optimized using losses like Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) or Mean Absolute Error (MAE) in the complex domain. However, a paradox has emerged: models that achieve the highest SI-SDR scores often sound worse to human ears and perform poorly in Automatic Speech Recognition (ASR) systems compared to magnitude-based models. This paper identifies a fundamental "compensation" mechanism between magnitude and phase that occurs during training as the root cause.

The Implicit Compensation Mechanism

The core of the investigation involves comparing "pure" losses against "magnitude-augmented" losses. Pure losses include $L_{RI}$ (Real/Imaginary components) and $L_{Wav}$ (Time-domain waveform), while magnitude-augmented losses add a term like $\| |\hat{S}| - |S| \|_1$ to the objective.

The Projection Effect

When a model cannot accurately estimate the phase of a signal, the mathematically "optimal" way to minimize a complex-domain loss is to project the target onto the estimated phase vector. This is known as the projection effect.

Vector representation of the projection effect where estimated magnitude compensates for phase error.

As illustrated in Figure 1, if the phase error is large, the model will shrink the estimated magnitude to minimize the overall complex-valued distance. If the phase error exceeds 90 degrees, the model may even predict a magnitude of zero. SI-SDR rewards this compensation because it measures sample-level alignment; a model that "shrinks" its magnitude to account for a phase shift will achieve a better SI-SDR than one that maintains the correct magnitude but has the same phase shift.

Visualizing the Compensation

To prove this theory, the authors utilized 2D histograms to visualize the relationship between phase error and magnitude ratio.

2D histograms showing the relationship between phase error and magnitude ratio across different training objectives.

In Figure 2, models trained with pure losses (RI, Wav) show a clear downward trend: as phase error increases, the magnitude ratio drops significantly. In contrast, adding a magnitude loss (RI+Mag, Wav+Mag) acts as a regularizer, preventing the model from shrinking the volume to hide phase errors. This constraint forces the model to prioritize magnitude accuracy, which is more critical for human perception and ASR performance.

Experimental Results

The study utilized benchmarks including WHAMR! (noisy-reverberant speech enhancement) and SMS-WSJ (2-speaker separation).

  • Quality vs. SDR Trade-off: Adding a magnitude loss (e.g., RI+Mag vs. RI) consistently led to a slight drop in SI-SDR (often ~0.5 to 1.4 dB) but a significant jump in PESQ (quality) and eSTOI (intelligibility). For example, in the SMS-WSJ task, RI+Mag improved PESQ from 1.97 to 2.21 compared to pure RI.
  • ASR Performance: On the SMS-WSJ dataset, the pure RI model resulted in a Word Error Rate (WER) of 42.26%. Adding a magnitude loss dropped the WER to 35.97%.
  • The Magnitude Advantage: Interestingly, the best ASR results often came from Magnitude Spectrogram Approximation (MSA), which ignores phase estimation entirely, achieving a WER of 32.84%. This suggests that for machine "listeners," a clean magnitude is far more important than a reconstructed phase.

Conclusion

This research demonstrates that human hearing and ASR systems are relatively robust to slight phase shifts but highly sensitive to magnitude distortions. By allowing models to compensate for phase errors through magnitude manipulation, standard end-to-end training objectives prioritize mathematical metrics over actual utility. Explicitly including a magnitude loss is essential for developing separation systems that serve real-world applications.

Summary updated

Abstract

Deep neural network (DNN) based end-to-end optimization in the complex time-frequency (T-F) domain or time domain has shown considerable potential in monaural speech separation. Many recent studies optimize loss functions defined solely in the time or complex domain, without including a loss on magnitude. Although such loss functions typically produce better scores if the evaluation metrics are objective time-domain metrics, they however produce worse scores on speech quality and intelligibility metrics and usually lead to worse speech recognition performance, compared with including a loss on magnitude. While this phenomenon has been experimentally observed by many studies, it is often not accurately explained and there lacks a thorough understanding on its fundamental cause. This paper provides a novel view from the perspective of the implicit compensation between estimated magnitude and phase. Analytical results based on monaural speech separation and robust automatic speech recognition (ASR) tasks in noisy-reverberant conditions support the validity of our view.
BibTeX
@article{wang2021compensation,
  author = {Zhong-Qiu Wang and Gordon Wichern and Jonathan Le Roux},
  title = {{On The Compensation Between Magnitude and Phase in Speech Separation}},
  journal = {IEEE Signal Processing Letters},
  year = {2021},
  url = {https://openreview.net/forum?id=rm283SSCST},
}
Figure