PixIT: Joint Training of Speaker Diarization and Speech Separation from Real-world Multi-speaker Recordings

Joonas Kalda, Clément Pagès, Ricard Marxer, Tanel Alumäe, Hervé Bredin

TL;DR

PixIT is a novel joint training framework that tackles the "synthetic data gap" in speech separation by learning directly from real-world multi-speaker recordings. By combining unsupervised Mixture Invariant Training (MixIT) with supervised Permutation Invariant Training (PIT) for diarization, the authors eliminate the common problem of "over-separation" and significantly improve the accuracy of automatic transcription in overlapping speech scenarios.

What Problem Does This Paper Address?

Speech separation—the "cocktail party problem"—is traditionally solved using supervised learning. However, supervised models require "clean" individual speaker tracks to create synthetic mixtures for training. In the real world (like noisy office meetings), we rarely have these clean references due to microphone cross-talk and complex acoustics, leading to models that perform poorly outside the lab.

Unsupervised methods like MixIT were designed to train on real recordings by mixing two existing mixtures together and asking the model to separate them back into their original components. While clever, MixIT often suffers from over-separation (splitting one person's voice into two different output channels) and struggles to maintain speaker consistency over long recordings. PixIT addresses this by using speaker diarization labels (who spoke when) as a "anchor" during training to keep the separation grounded and consistent.

Methods

The authors introduce a joint architecture called ToTaToNet, which performs both separation and diarization simultaneously.

  1. The Architecture: Based on the TasNet framework, it uses a 1D convolutional encoder/decoder and a DPRNN separator. Crucially, it incorporates pre-trained WavLM features, which provide rich phonetic and speaker context.
  2. Joint Training (PixIT): The model is trained on "Mixture of Mixtures" (MoM). It takes two real audio chunks, sums them, and tasks the model with separating the result.
  3. Separation Branch: Uses a MixIT loss to ensure the combined separated sources reconstruct the original mixtures.
  4. Diarization Branch: Uses a PIT loss against ground-truth diarization labels. This branch forces the model to align specific output channels with specific speakers, effectively "curing" the over-separation issue.
  5. Inference & Stitching: For long-form audio, the model processes sliding windows. It uses a clustering-based approach to "stitch" speakers across windows. A key post-processing step is leakage removal: if the diarization branch decides a speaker is inactive at a certain time, that channel is zeroed out to prevent "ghost" speech or noise from reaching the ASR system.

Datasets

The model was evaluated on two major real-world meeting corpora:

  • AMI: Approximately 100 hours of English meeting recordings.
  • AliMeeting: Approximately 120 hours of Mandarin Chinese recordings. This dataset is particularly challenging due to a much higher percentage of overlapping speech compared to AMI.

Key Insights and Ideas

  • Diarization as a Regularizer: The most significant "aha" moment is that you don't need clean source audio to train a great separator. You just need to know when people are talking. The diarization labels act as a constraint that prevents the unsupervised separation from hallucinating extra speakers.
  • The Power of Leakage Removal: The authors found that simply zeroing out the audio in a separated channel when the diarization model says the speaker is silent significantly boosts ASR performance. It removes the "cross-talk" that often confuses transcription models.
  • Zero Fine-tuning: PixIT improves transcription accuracy using off-the-shelf ASR models (like Whisper) without needing to fine-tune the ASR models themselves. This suggests the separated audio is high-quality and "natural" enough for general-purpose models.

Key Results

The authors measured success using cpWER (concatenated minimum permutation Word Error Rate), which accounts for speaker confusion.

  • AMI (English): PixIT achieved a relative reduction in cpWER of up to 17.2% using the Whisper large-v2 model compared to standard diarization-based attribution.
  • AliMeeting (Mandarin): The gains were even more dramatic due to the high overlap, with relative improvements exceeding 25% in character error rate (cpCER).
  • Diarization: Even though separation was the primary goal, the joint model achieved state-of-the-art (SOTA) results in Diarization Error Rate (DER) on both datasets when optimized specifically for that task.

Limitations and Future Work

  • Maximum Speaker Constraint: The current implementation sets a $K_{max}$ (maximum speakers per chunk) of 3. While sufficient for most 5-second windows in the tested datasets, it might struggle in extremely crowded environments.
  • Reliance on Labels: While PixIT doesn't need clean audio, it still requires diarization labels for training. Future work could explore using noisier, silver-standard labels generated by other models to make the process even more "unsupervised."
  • Computational Load: Processing multiple separated streams through ASR is more computationally expensive than processing a single mixture. Optimizing the pipeline for real-time use remains a challenge.

Summary updated

Abstract

A major drawback of supervised speech separation (SSep) systems is their reliance on synthetic data, leading to poor real-world generalization. Mixture invariant training (MixIT) was proposed as an unsupervised alternative that uses real recordings, yet struggles with overseparation and adapting to long-form audio. We introduce PixIT, a joint approach that combines permutation invariant training (PIT) for speaker diarization (SD) and MixIT for SSep. With a small extra requirement of needing SD labels, it solves the problem of overseparation and allows stitching local separated sources leveraging existing work on clustering-based neural SD. We measure the quality of the separated sources via applying automatic speech recognition (ASR) systems to them. PixIT boosts the performance of various ASR systems across two meeting corpora both in terms of the speaker-attributed and utterance-based word error rates while not requiring any fine-tuning.
BibTeX
@inproceedings{kalda2024pixit,
  author = {Joonas Kalda and Clément Pagès and Ricard Marxer and Tanel Alumäe and Hervé Bredin},
  title = {{PixIT: Joint Training of Speaker Diarization and Speech Separation from Real-world Multi-speaker Recordings}},
  booktitle = {Odyssey},
  year = {2024},
  doi = {10.21437/odyssey.2024-17},
}
Figure