Mutual Learning of Single- and Multi-Channel End-to-End Neural Diarization

Shota Horiguchi, Yuki Takashima, Shinji Watanabe, Paola Garcia

TL;DR

This paper proposes a mutual learning framework for speaker diarization that iteratively transfers knowledge between single-channel and multi-channel models. By alternating between knowledge distillation (multi-to-single) and fine-tuning (single-to-multi), the authors demonstrate that both models can "teach" each other, leading to significant performance gains by combining spectral speaker characteristics with spatial information.

What Problem Does This Paper Address?

Speaker diarization—the task of determining "who spoke when"—is notoriously difficult in noisy, reverberant environments with overlapping speech.

Researchers typically approach this in two ways:

  1. Single-channel (1-ch): These models rely entirely on spectral features (the unique "sound" of a person's voice).
  2. Multi-channel (M-ch): These models can leverage spatial information (where the sound is coming from) using microphone arrays.

While multi-channel models are generally more powerful, they can become "lazy," over-relying on spatial cues and ignoring useful speaker-specific voice characteristics. Conversely, single-channel models are limited by the lack of spatial context. The authors address the gap of how to effectively combine the strengths of both modalities without requiring complex, separate architectures.

Methods

The core innovation is a unified architecture and an iterative training loop.

1. The Unified Architecture: Co-attention

The authors use an attractor-based End-to-End Neural Diarization (EEND-EDA) model. To handle both 1-ch and M-ch inputs with the same set of parameters, they employ a co-attention mechanism.

The architecture of the encoders. Note how the co-attention encoder (b) generalizes the standard Transformer encoder (a).

As shown in Figure 2, the co-attention encoder is designed such that when the input is single-channel, it mathematically collapses into a standard Transformer self-attention layer. This allows a single model to be trained and evaluated on varying numbers of microphones seamlessly.

2. The Mutual Learning Loop

The training proceeds in two alternating steps:

  • Step 1: Multi-to-Single Distillation. A powerful multi-channel model acts as a "teacher." A single-channel "student" is trained to mimic the teacher's output (logits) using Mean Squared Error (MSE) loss. This forces the 1-ch model to learn better speaker representations guided by the teacher's spatial awareness.
  • Step 2: Single-to-Multi Fine-tuning. The improved 1-ch model is then used as a pre-trained starting point for a multi-channel model. Fine-tuning this model on multi-channel data ensures it doesn't ignore spectral features in favor of easy spatial ones.
Fig. 1: Mutual learning of single- and multi-channel EEND.

Datasets

The researchers evaluated their method on both simulated and real-world conversational data:

  • Simulated: A massive dataset (20,000 sessions) created from NIST SRE and Switchboard (SWBD) corpora, augmented with MUSAN noise and simulated room impulse responses.
  • Real Single-Channel: The well-known CALLHOME (CH-2spk) dataset and the Corpus of Spontaneous Japanese (CSJ).
  • Real Multi-Channel: Re-recorded versions of the CSJ dataset using distributed smartphone microphones to capture realistic spatial dynamics.

Key Insights and Ideas

  • Bi-directional Benefits: The "aha moment" is that the transfer isn't just one-way. While it's intuitive that a multi-channel teacher helps a single-channel student, the reverse is also true: single-channel pre-training acts as a regularizer for multi-channel models, preventing them from falling into local minima that rely too heavily on spatial geometry.
  • Logit Matching: Instead of using hard labels (who is speaking), the distillation uses the teacher's raw output scores (logits). This captures the teacher's "uncertainty" and provides a richer training signal for the student.
  • Iterative Improvement: The performance doesn't plateau after one round; a second round of mutual learning provided further gains for both models.

Key Results

The proposed method consistently reduced the Diarization Error Rate (DER) across the board:

  • Simulated Data:
    • The baseline 1-ch model had a DER of 4.11%. After distillation, this dropped to 3.34%, and eventually 3.08% after a second round.
  • The baseline 4-ch model improved from 2.32% to 2.08% through mutual learning.
  • Real Data (CALLHOME): The 1-ch model improved significantly, dropping from a baseline DER of 14.11% to 9.85% after two rounds of distillation.

Limitations and Future Work

  • Speaker Count: The experiments focused on two-speaker scenarios. While the underlying EEND-EDA model can handle an unknown number of speakers, the mutual learning dynamics for larger groups remain to be tested.
  • Out-of-Domain Robustness: In the most challenging "out-of-domain" real dataset (CSJ-dialog), the baseline multi-channel model actually slightly outperformed the mutually-learned one. This suggests that when spectral features are highly degraded or unfamiliar, spatial information remains the most reliable cue.
  • Future Directions: The authors suggest applying this mutual learning framework to other speech tasks, such as automatic speech recognition (ASR) and source separation.

Summary updated

Abstract

Due to the high performance of multi-channel speech processing, we can use the outputs from a multi-channel model as teacher labels when training a single-channel model with knowledge distillation. To the contrary, it is also known that single-channel speech data can benefit multi-channel models by mixing it with multi-channel speech data during training or by using it for model pretraining. This paper focuses on speaker diarization and proposes to conduct the above bi-directional knowledge transfer alternately. We first introduce an end-to-end neural diarization model that can handle both single- and multi-channel inputs. Using this model, we alternately conduct i) knowledge distillation from a multi-channel model to a single-channel model and ii) finetuning from the distilled single-channel model to a multi-channel model. Experimental results on two-speaker data show that the proposed method mutually improved single- and multi-channel speaker diarization performances.
BibTeX
@inproceedings{horiguchi2022mutual,
  author = {Shota Horiguchi and Yuki Takashima and Shinji Watanabe and Paola Garcia},
  title = {{Mutual Learning of Single- and Multi-Channel End-to-End Neural Diarization}},
  booktitle = {SLT},
  year = {2022},
  doi = {10.1109/slt54892.2023.10023388},
}
Figure