Abstract
PixIT is a novel joint training framework that simultaneously addresses speaker diarization (identifying who spoke when) and speech separation (isolating individual voices) using real-world recordings. By integrating unsupervised separation (MixIT) with supervised diarization (PIT), PixIT eliminates the reliance on synthetic training data and significantly enhances automatic transcription accuracy in overlapping speech scenarios.
Introduction
In speech processing, Speaker Diarization (SD) and Speech Separation (SSep) are vital for transcribing multi-speaker environments like meetings. Traditionally, these tasks are handled independently or in a cascaded manner. Supervised separation models often require clean individual audio tracks for training—data that is nearly impossible to obtain from real-world recordings due to microphone cross-talk. While synthetic mixtures are a common workaround, they often fail to generalize to real-world complexities. Unsupervised alternatives like Mixture Invariant Training (MixIT) suffer from "over-separation," where a single speaker's voice is split across multiple output channels.
Methods
Architecture: ToTaToNet
The core of the PixIT framework is ToTaToNet, an architecture based on TasNet but enhanced with pre-trained WavLM features to capture richer acoustic information.
The model consists of four primary components:
- Encoder: Transforms input audio into a high-dimensional representation.
- Masking Network: Predicts masks to isolate individual speaker sources.
- Diarization Branch: Predicts speaker activity (Voice Activity Detection) for each isolated source.
- Decoder: Reconstructs the masked representations back into time-domain audio.
Training Framework
The primary innovation lies in the joint loss function and the use of "Mixtures of Mixtures" (MoMs). By summing two real-world audio segments, the model is tasked with separating them back into their original (potentially already mixed) states using the MixIT loss.
Simultaneously, the model utilizes standard diarization labels—which are more accessible for real-world data than clean source audio—to ensure that separated sources align with actual speaker identities via Permutation Invariant Training (PIT). This joint approach forces the separation outputs to satisfy the diarization objective, effectively mitigating the over-separation problem common in unsupervised methods.
Datasets and Evaluation
PixIT was evaluated on two challenging real-world corpora using Single Distant Microphone (SDM) settings:
- AMI: Approximately 100 hours of English meeting data.
- AliMeeting: Approximately 120 hours of Mandarin Chinese data with high overlap density.
Performance was measured using off-the-shelf ASR models (e.g., OpenAI Whisper, NVIDIA NeMo) to assess how well the separation improved transcription quality.
Key Results
- Transcription Accuracy: On AMI, PixIT improved the concatenated minimum permutation Word Error Rate (cpWER) by up to 17.2% relative to a state-of-the-art diarization baseline (pyannote.audio 3.1).
- Mandarin Performance: On AliMeeting, the improvement exceeded 25% relative in character error rate (cpCER).
- Diarization Quality: PixIT achieved results competitive with or superior to specialized state-of-the-art diarization systems.
- Leakage Removal: Since diarization and separation are co-located in the same model, the system can zero out separation channels when the diarization branch detects silence, significantly reducing noise and cross-talk.
Conclusion
PixIT demonstrates that diarization labels can serve as effective weak supervision for speech separation, allowing models to be trained directly on real-world data. This "best-of-both-worlds" approach is now available as part of the open-source pyannote.audio library, bridging the gap between identifying speakers and transcribing their words.
Summary updated