Mixup-breakdown: A Consistency Training Method for Improving Generalization of Speech Separation Models

Max W. Y. Lam, Jun Wang, Dan Su, Dong Yu

TL;DR

Deep learning speech separation models often fail catastrophically when exposed to real-world, out-of-distribution background noise or music. This paper introduces Mixup-Breakdown Training (MBT), a novel semi-supervised learning method that uses a teacher-student framework to generate pseudo-labeled mixtures from abundant unlabeled data. By enforcing consistency between the teacher's "broken down" signals and the student's predictions on "mixed up" combinations, MBT drastically improves model generalization without adding significant computational overhead.

What Problem Does This Paper Address?

State-of-the-art speech separation models (like Conv-TasNet) perform incredibly well on benchmark datasets but suffer from severe overfitting. When these models encounter acoustic conditions during inference that differ even slightly from their training data—such as unseen background noise, different types of interfering speech, or music—their performance can abruptly collapse.

Collecting massive, perfectly labeled datasets covering every possible real-world acoustic scenario is prohibitively expensive and practically impossible. Meanwhile, vast amounts of unlabeled mixture audio exist but are ignored by standard supervised training pipelines. The authors tackle the challenge of utilizing this cheap, unlabeled data to make speech separation models robust to mismatched, out-of-domain interference.

Methods

The authors propose Mixup-Breakdown Training (MBT), a consistency-based semi-supervised learning (SSL) approach. It builds on the Mean Teacher framework and the popular mixup data augmentation technique, but adapts them specifically for source separation.

Here is how the pipeline works:

  1. Teacher-Student Setup: A teacher model's weights are updated continuously as an exponential moving average of a student model's weights.
  2. Breakdown: The teacher model takes an input mixture (which can be completely unlabeled) and separates it into its estimated constituent source signals.
  3. Mixup: These estimated source signals are then mathematically interpolated (mixed up) using a random ratio to create a new, pseudo-labeled mixture.
  4. Consistency Training: The student model is fed this new pseudo-mixture and penalized if its output doesn't match the teacher's interpolated source signals.

What is novel? Unlike standard mixup or Interpolation Consistency Training (ICT) which mix two completely different input samples together, MBT breaks down a single mixture into its sources and remixes those specific sources at different volume ratios. This creates highly effective, acoustically logical pseudo-labels for the separation task.

Datasets

  • WSJ0-2mix: The standard benchmark dataset used for training the base supervised models.
  • New Mismatch Datasets: To test generalization, the authors created three new evaluation sets by replacing the background speech in WSJ0-2mix with out-of-domain audio:
  • WSJ0-Libri: Uses clean speech from Librispeech (unseen interfering speech).
  • WSJ0-noise: Uses real-world environmental noises (office, restaurant, construction).
  • WSJ0-music: Uses a diverse 43-hour music dataset (classical, jazz, hip-hop).
  • WSJ0-multi: A combined unlabeled dataset of the above used for the semi-supervised training phase.

Key Insights and Ideas

  • Supervised models are incredibly brittle: The study reveals a shocking vulnerability in standard models trained via Empirical Risk Minimization (ERM). When tested on unseen noise or music, state-of-the-art Conv-TasNet models saw their performance plummet to near-useless levels.
  • Semi-supervised learning is a lifeline for robustness: Using unlabeled out-of-domain data during training allows the model to learn the underlying structure of new interference types without needing clean, ground-truth targets.
  • Remixing predictions works better than remixing inputs: MBT outperforms existing SSL techniques like Interpolation Consistency Training (ICT). By breaking down a mixture into sources and remixing those sources at new ratios, MBT creates more useful perturbations for the student model to learn from than simply blending two random audio files together.

Key Results

  • Supervised Baseline: On the standard WSJ0-2mix dataset, using MBT purely as an "online" data augmentation technique slightly improved the Conv-TasNet baseline from 15.3 to 15.5 SI-SNRi.
  • Mismatch Speech: On WSJ0-Libri, MBT achieved 13.95 SI-SNRi, outperforming standard training (13.56) and other SSL methods like Mean Teacher (13.81).
  • Mismatch Noise: Standard supervised models collapsed to a dismal 1.86 SI-SNRi. MBT trained with unlabeled noise data recovered performance to an impressive 13.52 SI-SNRi.
  • Mismatch Music: Standard models collapsed to 1.93 SI-SNRi. MBT achieved 15.95 SI-SNRi, a massive recovery and up to a 13.77% relative improvement over competing SSL methods like ICT.

Limitations and Future Work

  • Limitations: The current implementation of MBT focuses primarily on amplitude interpolation (essentially augmenting the Signal-to-Noise Ratio between the separated sources). It does not yet account for other complex acoustic variations.
  • Future Work: The authors plan to extend the Mixup-Breakdown methodology to include other types of perturbations, such as various audio distortions, speed alterations, and spatial location shifts (for multi-channel microphone array scenarios). This could further unlock the model's generalization capabilities across even more diverse acoustic environments.

Summary updated

Abstract

Deep-learning based speech separation models confront poor generalization problem that even the state-of-the-art models could abruptly fail when evaluating them in mismatch conditions. To address this problem, we propose an easy-to-implement yet effective consistency based semi-supervised learning (SSL) approach, namely Mixup-Breakdown training (MBT). It learns a teacher model to "breakdown" unlabeled inputs, and the estimated separations are interpolated to produce more useful pseudo "mixup" input-output pairs, on which the consistency regularization could apply for learning a student model. In our experiment, we evaluate MBT under various conditions with ascending degrees of mismatch, including unseen interfering speech, noise, and music, and compare MBT’s generalization capability against state-of-the-art supervised learning and SSL approaches. The result indicates that MBT significantly outperforms several strong baselines with up to 13.77% relative SI-SNRi improvement. Moreover, MBT only adds negligible computational overhead to standard training schemes.
BibTeX
@inproceedings{lam2020mixup,
  author = {Max W. Y. Lam and Jun Wang and Dan Su and Dong Yu},
  title = {{Mixup-breakdown: A Consistency Training Method for Improving Generalization of Speech Separation Models}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  doi = {10.1109/icassp40776.2020.9054719},
}
Figure