Mixture to Mixture: Leveraging Close-Talk Mixtures as Weak-Supervision for Speech Separation

Zhong-Qiu Wang

TL;DR

Mixture to Mixture (M2M) is a new weakly-supervised training framework that allows speech separation models to learn from "dirty" real-world data. By leveraging noisy close-talk microphone recordings (which contain crosstalk) as a guide, M2M achieves separation performance on far-field mic arrays that rivals fully supervised models trained on perfectly clean, synthetic data.

What Problem Does This Paper Address?

Deep learning has revolutionized speech separation—the "cocktail party problem" of isolating one voice from many. However, most state-of-the-art models rely on supervised learning, which requires "ground truth" clean audio for every speaker. In the real world, we can't record a clean version of one person speaking while others are talking in the same room. Consequently, researchers usually train models on synthetic data, which often fails to generalize to real-world acoustics.

Unsupervised learning (training on raw mixtures) exists but is often under-constrained and less performant. This paper identifies a "middle ground": in many recording scenarios (like the AMI or CHiME datasets), speakers wear close-talk microphones (headsets or lapels) while a distant microphone array also records the room. These close-talk recordings aren't "clean"—they contain "crosstalk" from other speakers and background noise—but they have a much higher Signal-to-Noise Ratio (SNR) for the target speaker. The author asks: Can we use these "noisy" close-talk mixtures as a weak supervision signal to train better far-field separators?

Methods

The core of the paper is the M2M training objective. Instead of forcing a Deep Neural Network (DNN) to produce a "clean" signal, M2M treats the DNN's output as an intermediate estimate.

  1. Input: The DNN takes in the far-field mixture.
  2. Intermediate Estimates: It produces an estimate for each speaker.
  3. Linear Filtering (FCP): To account for the fact that close-talk and far-field mics are in different locations and have different time alignments, the system uses Forward Convolutive Prediction (FCP). This is a set of learnable linear filters that map the DNN's intermediate estimates to what should be heard at every single microphone (both the far-field array and the close-talk mics).
  4. Mixture-Constraint (MC) Loss: The model is optimized so that the sum of these filtered speaker estimates reconstructs the actual recorded mixture at every microphone.

Because the close-talk mic has a much stronger signal for one specific speaker, the loss function "forces" the DNN to assign the correct voice to the correct output channel to satisfy the reconstruction at that specific microphone.

Datasets

The author introduces a new simulated benchmark: SMS-WSJ-FF-CT.

  • It is based on the popular SMS-WSJ (Spatialized Multi-Speaker Wall Street Journal) corpus, which simulates 2-speaker mixtures in reverberant rooms.
  • The "FF-CT" suffix stands for Far-Field + Close-Talk. The author added simulated close-talk microphones (placed 10–30 cm from each speaker) to the existing 6-microphone circular array setup.
  • This allows for a controlled environment where the "weak supervision" (noisy close-talk mics) can be tested against known "clean" ground truths for evaluation.

Key Insights and Ideas

  • Crosstalk is a Feature, Not a Bug: Previous methods struggled with close-talk mics because they aren't "clean." M2M embraces this by modeling the close-talk recording as just another mixture to be reconstructed.
  • Over-determined Constraints: By using $P$ far-field mics and $C$ close-talk mics to separate $C$ sources, the system becomes "over-determined." This extra information narrows down the possible solutions, preventing the model from getting stuck in the ambiguities that plague purely unsupervised methods.
  • Implicit Alignment: Because the FCP filters are computed per-utterance, the model automatically handles the micro-delays and acoustic differences between a headset mic and a distant array without needing manual time-alignment.

Key Results

The results show that M2M effectively bridges the gap between unsupervised and supervised learning:

  • Superior to Unsupervised: On a 6-channel far-field task, M2M achieved a 16.9 dB Scale-Invariant Signal-to-Distortion Ratio (SI-SDR), significantly outperforming the unsupervised UNSSOR baseline (14.7 dB).
  • Competitive with Supervised: Remarkably, M2M approached the performance of PIT (Permutation Invariant Training), which is fully supervised and requires clean audio (18.9 dB).
  • Monaural Success: Even when the model only had access to a single far-field microphone at test time, using the close-talk mics during training led to an SI-SDR of 12.7 dB, nearly matching the supervised monaural baseline of 13.7 dB.

Limitations and Future Work

  • Simulated vs. Real: While the results are promising, the study used simulated data with "weak noise." The author notes that the next step is evaluating M2M on real-recorded datasets like AMI, where noise and acoustic mismatches are more unpredictable.
  • Scaling Speakers: The current evaluation focused on 2-speaker mixtures. Scaling this to 3+ speakers or varying numbers of speakers remains to be tested.
  • Linearity Assumptions: The FCP filtering assumes a linear relationship between the source and the mic. In environments with heavy non-linear distortion or extremely high noise, these assumptions might need to be relaxed or replaced with more complex neural filters.

Summary updated

Abstract

We propose mixture to mixture (M2M) training, a weakly-supervised neural speech separation algorithm that leverages close-talk mixtures as a weak supervision for training discriminative models to separate far-field mixtures. Our idea is that, for a target speaker, its close-talk mixture has a much higher signal-to-noise ratio (SNR) of the target speaker than any far-field mixtures, and hence could be utilized to design a weak supervision for separation. To realize this, at each training step we feed a far-field mixture to a deep neural network (DNN) to produce an intermediate estimate for each speaker, and, for each of considered close-talk and far-field microphones, we linearly filter the DNN estimates and optimize a loss so that the filtered estimates of all the speakers can sum up to the mixture captured by each of the considered microphones. Evaluation results on a 2-speaker separation task in simulated reverberant conditions show that M2M can effectively leverage close-talk mixtures as a weak supervision for separating far-field mixtures.
BibTeX
@article{wang2024mixture,
  author = {Zhong-Qiu Wang},
  title = {{Mixture to Mixture: Leveraging Close-Talk Mixtures as Weak-Supervision for Speech Separation}},
  journal = {IEEE Signal Processing Letters},
  year = {2024},
  url = {https://openreview.net/forum?id=IRF6oPFVZm},
}

Loading related content...

Figure