Abstract
This paper presents a novel approach to speaker diarization—the process of partitioning an audio stream into segments based on speaker identity—by leveraging a deep learning architecture to learn robust speaker embeddings. Unlike traditional methods that rely on hand-crafted spectral features and modular pipelines, we propose a unified Deep Recurrent Convolutional Neural Network (R-CNN) trained directly on magnitude spectrograms. To evaluate our approach, we introduce a new, publicly available dataset of over six hours of annotated broadcast material (BNV). Our results across four benchmark datasets demonstrate that the R-CNN architecture significantly outperforms state-of-the-art baselines, reducing the Diarization Error Rate (DER) by over 30%.
Introduction
Speaker diarization, often summarized as the task of determining "who spoke when," is a critical component in automated speech processing. It serves as a foundational step for meeting transcription, broadcast indexing, and multi-party conversation analysis. Historically, this task has been addressed through a pipeline of independent modules: Voice Activity Detection (VAD), Speaker Change Detection (SCD), and Speaker Identification. These systems typically utilize "i-vectors" and hand-crafted features like Mel-Frequency Cepstral Coefficients (MFCCs).
However, these modular approaches suffer from cascading errors—where a failure in change detection propagates through the identification stage—and struggle with the nuances of natural speech, such as overlapping voices and varying acoustic environments. This research addresses these limitations by replacing manual feature engineering with learned embeddings derived from a deep neural network designed to capture both local spectral patterns and long-term temporal dependencies.
Methodology
Feature Extraction: The Constant Q Transform
While log-Mel spectrograms are standard in audio processing, this study finds that the Constant Q Transform (CQT) provides a superior input representation for speaker diarization. CQT offers better frequency resolution for vocal signals, which is particularly beneficial when the network must distinguish between subtle differences in speaker timbre. The magnitude spectrograms derived from CQT serve as the primary input to the neural architecture.
R-CNN Architecture
The proposed model is a Deep Recurrent Convolutional Neural Network consisting of 11 learnable layers:
- Convolutional Blocks: Four blocks designed to extract local acoustic patterns from the spectrogram.
- Recurrent Layers: Two Gated Recurrent Units (GRUs) that aggregate temporal information across the audio sequence. The inclusion of recurrence is vital, as it allows the model to maintain a "memory" of the speaker's vocal characteristics over time.
- Fully Connected Layer: A final layer that produces the high-level speaker embeddings.
Diarization Pipeline
The system operates in three phases. First, the R-CNN is trained in a supervised manner on a large-scale dataset to classify known speakers. Second, once the model is optimized, the final layer's activations are extracted as numerical "signatures" or embeddings. Finally, for unseen audio, the system uses cosine similarity to cluster these embeddings, effectively partitioning the audio into distinct speaker segments without requiring prior knowledge of the individuals involved.
Experimental Setup and Datasets
The model was evaluated using three established datasets and one new benchmark:
- AMI Meeting Corpus: 100 hours of office-based interactions.
- ICSI Meeting Corpus: 72 hours of academic discussion.
- YouTube Speakers Corpus (YT): 550 hours of diverse, real-world audio.
- Broadcast News Videos (BNV): A new dataset introduced by the authors containing over 6 hours of annotated content from networks like CNN and the BBC. This dataset is specifically designed to test generalization in noisy, "in-the-wild" broadcast environments.
Results and Discussion
Performance Gains
The R-CNN approach demonstrated a substantial improvement over the industry-standard LIUM baseline ( a Gaussian Mixture Model-based system). Across all tested datasets, the proposed method achieved a reduction in Diarization Error Rate (DER) of more than 30%. On the challenging BNV dataset, the R-CNN with CQT features reached a DER of 19.6%, compared to 28.5% for the baseline.
The Importance of Recurrence and CQT
A key finding of the study is the synergy between convolutional and recurrent layers. The R-CNN outperformed vanilla CNN architectures by approximately 12%, confirming that temporal context is essential for distinguishing between speakers. Furthermore, the choice of input feature proved critical; the CQT-based model consistently outperformed those using Gammatone or standard log-Mel features.
Generalization
Despite being trained on a finite set of speakers, the learned embeddings proved highly robust. The model successfully diarized audio from the BNV dataset containing entirely "unseen" speakers, suggesting that the R-CNN learns fundamental vocal characteristics rather than simply memorizing specific identities.
Conclusion
By shifting from hand-crafted features to learned embeddings via a Deep R-CNN, this research sets a new benchmark for speaker diarization. The 30% reduction in DER represents a significant step toward more reliable automated transcription and analysis. Future work will focus on improving robustness in extremely noisy environments and developing source separation techniques to better handle overlapping speech. The release of the BNV dataset further provides the research community with a rigorous tool for evaluating future diarization models.
Summary updated