Abstract
Speaker diarization—the process of partitioning an audio stream into segments according to speaker identity—benefits significantly from multi-channel audio, which provides spatial cues for separation. However, multi-channel models often become overly dependent on these spatial features, neglecting the robust spectral characteristics of individual voices. This paper presents a mutual learning framework designed to create a synergistic relationship between single-channel and multi-channel End-to-End Neural Diarization (EEND) models. By iteratively distilling knowledge from a multi-channel "teacher" into a single-channel "student" and subsequently using the student to re-initialize the teacher, the proposed method achieves substantial performance gains across both modalities on simulated and real-world datasets.
Introduction
The task of "who spoke when" is a fundamental component of speech processing. While multi-channel systems leverage spatial information to distinguish speakers, single-channel systems must rely entirely on spectral characteristics. This creates a performance gap where multi-channel systems generally outperform single-channel ones but often fail to learn the deep voice-identity features that single-channel models are forced to master.
This research addresses this imbalance by asking whether the spatial "wisdom" of multi-channel models can be transferred to single-channel systems, and whether the resulting robust spectral features can, in turn, improve multi-channel performance. The authors propose a bi-directional knowledge transfer process that breaks the spatial dependency of multi-channel models and enhances the accuracy of single-channel diarization.
Methodology
Unified Architecture
The framework utilizes a co-attention-based EEND architecture. The co-attention mechanism is particularly effective because it remains invariant to the number of microphones. When provided with a single channel, the model functions as a standard Transformer; when multiple channels are available, it processes the cross-channel relationships to extract spatial information. This symmetry allows for direct weight transfer between single- and multi-channel tasks without requiring specialized mapping layers.
The Mutual Learning Loop
The core of the proposal is a two-step iterative process:
- Multi-to-Single (Knowledge Distillation): A pre-trained multi-channel model acts as a teacher. It provides "soft labels" (logits) to a single-channel student model. Through Knowledge Distillation (KD), the student learns to mimic the superior separation capabilities of the multi-channel system, effectively learning to approximate spatial separation using only spectral data.
- Single-to-Multi (Finetuning): The distilled single-channel model is then used as the initialization for a multi-channel model, which is finetuned on multi-channel data. Because the model has already been regularized to prioritize robust speaker characteristics during the single-channel phase, it avoids the local minima of over-relying on spatial cues.
Loss Function
To account for the fact that speaker ordering in the output is arbitrary, the authors employ a permutation-free knowledge distillation loss. This uses Mean Squared Error (MSE) between the teacher and student logits, ensuring the grouping remains correct regardless of the specific output index assigned to a speaker.
Experimental Setup
Datasets
The framework was evaluated using a combination of simulated and real-world data:
- Simulated Data: Mixtures of Speech Recognition Evaluation (SRE) and Switchboard (SWBD) data, augmented with MUSAN noise and simulated room impulse responses.
- Real Single-Channel Data: The 2-speaker subset of the CALLHOME dataset and the Corpus of Spontaneous Japanese (CSJ).
- Real Multi-Channel Data: Re-recorded versions of the CSJ (CSJ-train, CSJ-eval, and CSJ-dialog) captured using distributed smartphones to simulate realistic multi-microphone environments.
Results and Discussion
Experimental results on two-speaker data demonstrated a clear "win-win" for both model types. On simulated data, the baseline single-channel Diarization Error Rate (DER) of 4.11% was reduced to 3.08% after distillation. Simultaneously, the multi-channel model improved from a 2.32% DER to 2.08% after student-based initialization.
On the real-world CALLHOME dataset, the method showed significant robustness, reducing the DER from 14.11% to 9.85%. The results on the CSJ dataset further confirmed that the learned features generalize well to out-of-domain environments, proving that the mutual learning process does not simply overfit to simulation parameters.
One notable observation occurred in the "CSJ-dialog" test, which involves complex real-world conversations. In this specific case, the baseline multi-channel model occasionally outperformed the mutually-learned version. The authors hypothesize that in extremely complex acoustic environments, the heavy reliance on spatial information is a necessary feature rather than a limitation. However, for the majority of scenarios, the mutual learning framework provided superior results.
Conclusion
This paper demonstrates that single-channel training acts as a powerful regularizer for multi-channel diarization models. By forcing the network to learn what a person sounds like rather than just where they are, the mutual learning framework improves the accuracy and robustness of both single- and multi-channel systems. Future work will explore extending this framework to handle an unknown number of speakers and applying the strategy to related tasks such as speech separation and automatic speech recognition (ASR).
Summary updated