TL;DR
This paper introduces DiaCorrect, a novel end-to-end framework that refines initial speaker diarization results by jointly modeling acoustic features and speaker activity sequences. By adapting techniques from ASR error correction, DiaCorrect significantly reduces diarization errors without relying on complex multi-stage pipelines or pre-trained speaker embedding extractors.
What Problem Does This Paper Address?
Speaker diarization, the task of identifying "who spoke when" in an audio recording, is crucial for many applications like meeting transcription and media retrieval. Traditional methods often involve complex multi-stage pipelines, which can be cumbersome and computationally expensive. While end-to-end neural diarization (EEND) models have shown promise, they can still produce errors. Existing error correction techniques, inspired by Automatic Speech Recognition (ASR), are challenging to apply directly to speaker diarization because the output (speaker activity) is different from ASR's text sequences. This paper aims to bridge this gap by proposing a simpler, end-to-end approach to correct initial diarization errors.
Methods
The core of DiaCorrect is an end-to-end framework that takes the original acoustic features (X) and an initial speaker activity sequence (Z) as input and outputs a refined speaker activity sequence (Ž). This is achieved through a parallel encoder-decoder architecture:
- SAS Encoder: This component models the statistics of the initial diarization results. It can accept two types of inputs:
- SASr: A binary (0/1) speaker activity sequence derived from a ground-truth RTTM file.
- SASp: A probability sequence representing speaker activity, directly obtained from a pre-trained model's output.
- Speech Encoder: This encoder processes the original acoustic features (Log-Mel spectrograms) to capture relevant audio information.
- Decoder: A transformer-based decoder integrates the outputs from both encoders to predict the refined speaker activity.
The model is trained using a permutation-invariant training (PIT) approach and a binary cross-entropy loss, similar to EEND models, to handle label ambiguity. A key feature is the ability to perform iterative inference, where the refined output from one pass can be fed back as input for further refinement.
Datasets
The experiments were conducted on a simulated 2-speaker meeting-like dataset derived from the LibriSpeech-100 corpus. The simulation procedure involved adding reverberation and controlling overlap ratios. The dataset was split into training, development, and testing sets.
Key Insights and Ideas
- Adapting ASR Error Correction: The paper successfully demonstrates that error correction techniques, proven effective in ASR, can be adapted for speaker diarization, opening new avenues for improving diarization performance.
- End-to-End Simplicity: DiaCorrect offers a significant advantage by consolidating error correction into a single, end-to-end model, eliminating the need for complex multi-stage pipelines and pre-trained speaker embedding extractors.
- Value of Probabilistic Speaker Activity (SASp): The study highlights that using probabilistic speaker activity sequences (SASp) as input to the error correction model yields substantially better results than using binary sequences (SASr). This is attributed to SASp providing richer information about the likelihood of a speaker's activity at any given time.
- Iterative Refinement: The ability to iteratively refine the diarization output proves beneficial, leading to further performance improvements, especially in reducing missed speech segments.
Key Results
- DiaCorrect, when using the SASp input and two iterations of refinement, achieved a Diarization Error Rate (DER) of 4.63% and a Jaccard Error Rate (JER) of 10.25% on the test set.
- This represents a significant reduction of over 62.4% in DER compared to the SA-EEND baseline (12.31%).
- The proposed 2-D convolution network for the speech encoder was found to be more effective than a single linear layer in capturing speech characteristics.
- Visualizations showed that DiaCorrect effectively corrected unexpected silent areas (reducing MISS errors) and speaker confusion (reducing SC errors) present in the baseline output.
Limitations and Future Work
- Dataset Specificity: The current evaluation is limited to a 2-speaker simulated dataset from LibriSpeech. The authors acknowledge that generalizing DiaCorrect to real-world scenarios with varying numbers of speakers and acoustic conditions is a crucial next step.
- Future Work: The authors plan to generalize DiaCorrect to different speaker diarization tasks, implying an interest in extending its applicability beyond the current experimental setup.
Summary updated