On the Compensation Between Magnitude and Phase in Speech Separation

Zhong-Qiu Wang, Gordon Wichern, Jonathan Le Roux

TL;DR

In speech separation, deep learning models trained solely on time-domain or complex-domain losses often "cheat" by distorting the signal's magnitude to compensate for inaccurate phase estimates. This paper demonstrates that adding an explicit magnitude loss prevents this mathematical trade-off, leading to significantly better speech quality (PESQ), intelligibility (STOI), and word error rates (WER), even if it slightly lowers the standard SI-SDR score.

What Problem Does This Paper Address?

Recent breakthroughs in speech separation have moved toward "end-to-end" models that operate directly on waveforms or complex-valued spectrograms. These models are typically optimized using losses like Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) or Mean Absolute Error (MAE) in the complex domain.

However, researchers noticed a paradox: models that achieved the highest SI-SDR scores often sounded worse to human ears and performed poorly in Automatic Speech Recognition (ASR) systems compared to older, magnitude-based models. The authors of this paper set out to explain why this happens, identifying a fundamental "compensation" mechanism between magnitude and phase that occurs during training.

Methods

The authors compare several training objectives across two main architectures: DenseUNet-TCN (for complex spectral mapping) and Conv-TasNet (for time-domain separation).

The core of their investigation involves comparing "pure" losses against "magnitude-augmented" losses:

  • Pure Losses: $L_{RI}$ (Real/Imaginary components) and $L_{Wav}$ (Time-domain waveform).
  • Magnitude-Augmented: Adding a term like $\| |\hat{S}| - |S| \|_1$ to the pure losses.
  • Baselines: They also look at Magnitude Spectrogram Approximation (MSA), which ignores phase estimation entirely, and Phase-sensitive Spectrogram Approximation (PSA).

To prove their theory, they use 2D histograms to visualize the relationship between phase error and magnitude ratio, showing exactly how the model shrinks or grows the estimated magnitude to minimize the overall complex-valued distance when it can't get the phase right.

Datasets

The study utilizes two major benchmarks:

  1. WHAMR!: A version of the Wall Street Journal (WSJ) corpus processed with room impulse responses and ambient noise. The authors tailored this for a noisy-reverberant speech enhancement task.
  2. SMS-WSJ: A spatialized multi-speaker WSJ dataset used here for 2-speaker separation in reverberant conditions.
  3. ASR Backend: A standard Kaldi-based speech recognition system was used to evaluate how separation performance translates to real-world utility.

Key Insights and Ideas

  • The "Projection" Effect: When a model cannot accurately estimate the phase of a signal, the mathematically "optimal" way to minimize a complex-domain loss is to project the target onto the estimated phase vector. If the phase error is large (e.g., > 90 degrees), the model will actually predict a magnitude of zero to minimize the error.
  • SI-SDR is a Double-Edged Sword: SI-SDR rewards this compensation because it measures sample-level alignment. A model that "shrinks" its magnitude to account for a phase shift will get a better SI-SDR than one that keeps the correct magnitude but has the same phase shift.
  • Human Perception vs. Math: Human hearing (and ASR systems) are relatively robust to slight phase shifts but very sensitive to magnitude distortions. By allowing the model to compensate, we are essentially training it to prioritize a metric (SI-SDR) over actual speech quality.
  • Magnitude Loss as a Regularizer: Adding an explicit magnitude loss acts as a constraint. It tells the model, "You aren't allowed to shrink the volume just to hide your phase errors."

Key Results

  • Quality vs. SDR Trade-off: In almost every test, adding a magnitude loss (e.g., RI+Mag vs. RI) led to a slight drop in SI-SDR (often ~0.5 to 1.0 dB) but a significant jump in PESQ (quality) and eSTOI (intelligibility).
  • ASR Performance: On the SMS-WSJ dataset, the pure RI model had a Word Error Rate (WER) of 42.26%. Adding a magnitude loss dropped the WER to 35.97%.
  • The Winner for ASR: Interestingly, the best ASR results often came from models that didn't try to reconstruct the phase at all (MSA), achieving a WER of 32.84%. This suggests that for machine "listeners," a clean magnitude is far more important than a reconstructed phase.

Limitations and Future Work

The authors note that the effectiveness of these losses can be sensitive to hyperparameters like window length—for instance, the small windows used in Conv-TasNet (5ms) showed different behavior than the larger windows (32ms) used in STFT-based models.

Future work could involve more sophisticated ways to balance these losses automatically. The authors also suggest that this insight should guide the design of "joint" systems, such as those performing sound event detection or ASR, where the separation front-end should be explicitly trained with magnitude-aware losses to better serve the downstream "back-end" model.

Summary updated

Abstract

Deep neural network (DNN) based end-to-end optimization in the complex time-frequency (T-F) domain or time domain has shown considerable potential in monaural speech separation. Many recent studies optimize loss functions defined solely in the time or complex domain, without including a loss on magnitude. Although such loss functions typically produce better scores if the evaluation metrics are objective time-domain metrics, they however produce worse scores on speech quality and intelligibility metrics and usually lead to worse speech recognition performance, compared with including a loss on magnitude. While this phenomenon has been experimentally observed by many studies, it is often not accurately explained and there lacks a thorough understanding on its fundamental cause. This letter provides a novel view from the perspective of the implicit compensation between estimated magnitude and phase. Analytical results based on monaural speech separation and robust automatic speech recognition (ASR) tasks in noisy-reverberant conditions support the validity of our view.
BibTeX
@article{wang2021compensation,
  author = {Zhong-Qiu Wang and Gordon Wichern and Jonathan Le Roux},
  title = {{On the Compensation Between Magnitude and Phase in Speech Separation}},
  journal = {IEEE Signal Processing Letters},
  year = {2021},
  url = {https://openreview.net/forum?id=rm283SSCST},
}

Loading related content...

Figure