PixIT: Joint Training of Speaker Diarization and Speech Separation from Real-world Multi-speaker Recordings

Joonas Kalda, Clément Pagés, Ricard Marxer, Tanel Alumäe, Hervé Bredin

Abstract

PixIT is a novel joint training framework that simultaneously addresses speaker diarization (identifying who spoke when) and speech separation (isolating individual voices) using real-world recordings. By integrating unsupervised separation (MixIT) with supervised diarization (PIT), PixIT eliminates the reliance on synthetic training data and significantly enhances automatic transcription accuracy in overlapping speech scenarios.

Introduction

In speech processing, Speaker Diarization (SD) and Speech Separation (SSep) are vital for transcribing multi-speaker environments like meetings. Traditionally, these tasks are handled independently or in a cascaded manner. Supervised separation models often require clean individual audio tracks for training—data that is nearly impossible to obtain from real-world recordings due to microphone cross-talk. While synthetic mixtures are a common workaround, they often fail to generalize to real-world complexities. Unsupervised alternatives like Mixture Invariant Training (MixIT) suffer from "over-separation," where a single speaker's voice is split across multiple output channels.

Methods

Architecture: ToTaToNet

The core of the PixIT framework is ToTaToNet, an architecture based on TasNet but enhanced with pre-trained WavLM features to capture richer acoustic information.

ToTaToNet Architecture

The model consists of four primary components:

  1. Encoder: Transforms input audio into a high-dimensional representation.
  2. Masking Network: Predicts masks to isolate individual speaker sources.
  3. Diarization Branch: Predicts speaker activity (Voice Activity Detection) for each isolated source.
  4. Decoder: Reconstructs the masked representations back into time-domain audio.

Training Framework

The primary innovation lies in the joint loss function and the use of "Mixtures of Mixtures" (MoMs). By summing two real-world audio segments, the model is tasked with separating them back into their original (potentially already mixed) states using the MixIT loss.

PixIT Training Framework

Simultaneously, the model utilizes standard diarization labels—which are more accessible for real-world data than clean source audio—to ensure that separated sources align with actual speaker identities via Permutation Invariant Training (PIT). This joint approach forces the separation outputs to satisfy the diarization objective, effectively mitigating the over-separation problem common in unsupervised methods.

Datasets and Evaluation

PixIT was evaluated on two challenging real-world corpora using Single Distant Microphone (SDM) settings:

  • AMI: Approximately 100 hours of English meeting data.
  • AliMeeting: Approximately 120 hours of Mandarin Chinese data with high overlap density.

Performance was measured using off-the-shelf ASR models (e.g., OpenAI Whisper, NVIDIA NeMo) to assess how well the separation improved transcription quality.

Key Results

  • Transcription Accuracy: On AMI, PixIT improved the concatenated minimum permutation Word Error Rate (cpWER) by up to 17.2% relative to a state-of-the-art diarization baseline (pyannote.audio 3.1).
  • Mandarin Performance: On AliMeeting, the improvement exceeded 25% relative in character error rate (cpCER).
  • Diarization Quality: PixIT achieved results competitive with or superior to specialized state-of-the-art diarization systems.
  • Leakage Removal: Since diarization and separation are co-located in the same model, the system can zero out separation channels when the diarization branch detects silence, significantly reducing noise and cross-talk.

Conclusion

PixIT demonstrates that diarization labels can serve as effective weak supervision for speech separation, allowing models to be trained directly on real-world data. This "best-of-both-worlds" approach is now available as part of the open-source pyannote.audio library, bridging the gap between identifying speakers and transcribing their words.

Summary updated

Abstract

A major drawback of supervised speech separation (SSep) systems is their reliance on synthetic data, leading to poor real-world generalization. Mixture invariant training (MixIT) was proposed as an unsupervised alternative that uses real recordings, yet struggles with overseparation and adapting to long-form audio. We introduce PixIT, a joint approach that combines permutation invariant training (PIT) for speaker diarization (SD) and MixIT for SSep. With a small extra requirement of needing SD labels, it solves the problem of overseparation and allows stitching local separated sources leveraging existing work on clustering-based neural SD. We measure the quality of the separated sources via applying automatic speech recognition (ASR) systems to them. PixIT boosts the performance of various ASR systems across two meeting corpora both in terms of the speaker-attributed and utterance-based word error rates while not requiring any fine-tuning.
BibTeX
@inproceedings{kalda2024pixit,
  author = {Joonas Kalda and Clément Pagés and Ricard Marxer and Tanel Alumäe and Hervé Bredin},
  title = {{PixIT: Joint Training of Speaker Diarization and Speech Separation from Real-world Multi-speaker Recordings}},
  booktitle = {Odyssey},
  year = {2024},
  doi = {10.21437/odyssey.2024-17},
}
Figure