SuperM2M: Supervised and Mixture-to-Mixture Co-Learning for Speech Enhancement and Noise-Robust ASR

Zhong-Qiu WangNeural Networks 2024

TL;DR

SuperM2M is a novel co-learning framework that solves the "generalization gap" in speech enhancement by training on both simulated and real-recorded data simultaneously. By adapting "mixture-to-mixture" (M2M) training as a form of weak supervision, the model learns the characteristics of real-world acoustics without needing impossible-to-get "clean" ground truth for real recordings, achieving new state-of-the-art results on the CHiME-4 benchmark.

What Problem Does This Paper Address?

The "bread and butter" of modern speech enhancement is supervised learning: you take clean speech, digitally add noise and reverberation, and train a Deep Neural Network (DNN) to reverse the process. While this works brilliantly on paper, these models often crumble when faced with real-world recordings.

The reason is the simulation gap. Digital simulations struggle to capture the messy reality of hardware: microphone synchronization errors, non-linear distortions, and complex, non-stationary noise environments (like a busy street or a rumbling bus). To fix this, we need to train on real data. However, training on real data is a "chicken and egg" problem: to train a supervised model, you need the clean version of the speech, but in a real-world recording, the clean version doesn't exist.

Methods

The authors introduce SuperM2M, which combines two training strategies in a single pipeline:

  1. Supervised Learning (on Simulated Data): Standard training using paired "noisy-simulated" and "clean-original" speech. This teaches the model the fundamental "concept" of what human speech sounds like.
  2. Mixture-to-Mixture (M2M) Learning (on Real Data): This is the core innovation. In many datasets (like CHiME-4), speakers wear a "close-talk" microphone (like a lapel mic) while being recorded by "far-field" microphones (like a tablet on a table). The close-talk mic isn't perfectly clean, but it has a much higher signal-to-noise ratio.
  3. The M2M approach treats the far-field mixture and the close-talk mixture as two different views of the same sources.
  4. The DNN predicts speech and noise estimates from the far-field input.
  5. A Mixture-Constraint Loss then checks if these estimates can be linearly filtered to reconstruct both the far-field and the close-talk recordings.

Key Technical Addition: Because real hardware often has timing lags between different microphones, the authors implemented an on-the-fly filter tap estimation algorithm. This allows the model to "re-align" the close-talk and far-field signals during training, preventing synchronization errors from ruining the loss calculation.

Datasets

The primary benchmark is CHiME-4, a famous dataset for noise-robust ASR. It includes:

  • Simulated Data (SIMU): ~15 hours of speech mixed with recorded noises.
  • Real Data (REAL): ~3 hours of actual recordings in four environments: Cafe, Bus, Pedestrian area, and Street.
  • Microphone Setup: A 6-channel tablet array and a 1-channel close-talk microphone.

Key Insights and Ideas

  • Noise as a Unified Source: Instead of trying to model every individual noise source (cars, wind, chatter), M2M models all non-speech signals as a single, combined "noise source." This simplifies the blind deconvolution problem into a two-source separation task.
  • The Power of Co-Learning: Unsupervised or weakly supervised models often suffer from "permutation" problems (where the model swaps speech and noise) or "frequency ambiguity." By alternating mini-batches of simulated (supervised) and real (M2M) data, the supervised batches act as an anchor, keeping the model's definitions of "speech" and "noise" consistent while it learns the "texture" of real-world data from the M2M batches.
  • Weak Supervision is Enough: You don't need a perfectly clean reference to improve a model. The "hint" provided by a noisy close-talk microphone is sufficient to guide the model toward better generalization.

Key Results

The authors evaluated the system using two backend ASR models: OpenAI's Whisper (Large v2) and a state-of-the-art ESPnet model.

  • Generalization: On the CHiME-4 REAL test set, purely supervised models showed massive performance degradation (WERs as high as 53-79%). SuperM2M maintained high performance, achieving a WER of 4.04% with Whisper.
  • New State-of-the-Art: Using the ESPnet backend on the 6-channel REAL test set, SuperM2M reached a Word Error Rate (WER) of 1.58%. This outperforms previous leading methods like IRIS (3.92%) and MultiIRIS (1.77%).
  • Single-Channel Success: Even when using only one microphone for enhancement (the most difficult task), SuperM2M achieved a 2.40% WER on real data, significantly better than the unprocessed baseline (4.47%).

Limitations and Future Work

  • Close-Talk Dependency: While the authors show the model can function without close-talk mixtures (reverting to an unsupervised mode called UNSSOR), the best performance gains rely on having those paired recordings during the training phase.
  • Computational Complexity: The M2M loss involves solving a quadratic optimization problem (FCP) for filter estimation at every training step, which increases training time compared to simple supervised learning.
  • Future Directions: The authors suggest extending this to "cocktail party" scenarios with multiple overlapping speakers and exploring more complex, non-linear filter models to better represent real-world microphone distortions.

Summary updated

Abstract

The current dominant approach for neural speech enhancement is based on supervised learning by using simulated training data. The trained models, however, often exhibit limited generalizability to real-recorded data. To address this, this paper investigates training enhancement models directly on real target-domain data. We propose to adapt mixture-to-mixture (M2M) training, originally designed for speaker separation, for speech enhancement, by modeling multi-source noise signals as a single, combined source. In addition, we propose a co-learning algorithm that improves M2M with the help of supervised algorithms. When paired close-talk and far-field mixtures are available for training, M2M realizes speech enhancement by training a deep neural network (DNN) to produce speech and noise estimates in a way such that they can be linearly filtered to reconstruct the close-talk and far-field mixtures. This way, the DNN can be trained directly on real mixtures, and can leverage close-talk and far-field mixtures as a weak supervision to enhance far-field mixtures. To improve M2M, we combine it with supervised approaches to co-train the DNN, where mini-batches of real close-talk and far-field mixture pairs and mini-batches of simulated mixture and clean speech pairs are alternately fed to the DNN, and the loss functions are respectively (a) the mixture reconstruction loss on the real close-talk and far-field mixtures and (b) the regular enhancement loss on the simulated clean speech and noise. We find that, this way, the DNN can learn from real and simulated data to achieve better generalization to real data. We name this algorithm SuperM2M (supervised and mixture-to-mixture co-learning). Evaluation results on the CHiME-4 dataset show its effectiveness and potential.
BibTeX
@article{wang2024superm,
  author = {Zhong-Qiu Wang},
  title = {{SuperM2M: Supervised and Mixture-to-Mixture Co-Learning for Speech Enhancement and Noise-Robust ASR}},
  journal = {Neural Networks},
  year = {2024},
}
Figure