Speaker Diarization Using Deep Recurrent Convolutional Neural Networks for Speaker Embeddings

Pawel Cyrta, Tomasz Trzcinski, Wojciech StokowiecISAT 2017 · DOI · OpenReview

Abstract

This paper presents a novel approach to speaker diarization—the process of partitioning an audio stream into segments based on speaker identity—by leveraging a deep learning architecture to learn robust speaker embeddings. Unlike traditional methods that rely on hand-crafted spectral features and modular pipelines, we propose a unified Deep Recurrent Convolutional Neural Network (R-CNN) trained directly on magnitude spectrograms. To evaluate our approach, we introduce a new, publicly available dataset of over six hours of annotated broadcast material (BNV). Our results across four benchmark datasets demonstrate that the R-CNN architecture significantly outperforms state-of-the-art baselines, reducing the Diarization Error Rate (DER) by over 30%.

Introduction

Speaker diarization, often summarized as the task of determining "who spoke when," is a critical component in automated speech processing. It serves as a foundational step for meeting transcription, broadcast indexing, and multi-party conversation analysis. Historically, this task has been addressed through a pipeline of independent modules: Voice Activity Detection (VAD), Speaker Change Detection (SCD), and Speaker Identification. These systems typically utilize "i-vectors" and hand-crafted features like Mel-Frequency Cepstral Coefficients (MFCCs).

However, these modular approaches suffer from cascading errors—where a failure in change detection propagates through the identification stage—and struggle with the nuances of natural speech, such as overlapping voices and varying acoustic environments. This research addresses these limitations by replacing manual feature engineering with learned embeddings derived from a deep neural network designed to capture both local spectral patterns and long-term temporal dependencies.

Methodology

Feature Extraction: The Constant Q Transform

While log-Mel spectrograms are standard in audio processing, this study finds that the Constant Q Transform (CQT) provides a superior input representation for speaker diarization. CQT offers better frequency resolution for vocal signals, which is particularly beneficial when the network must distinguish between subtle differences in speaker timbre. The magnitude spectrograms derived from CQT serve as the primary input to the neural architecture.

R-CNN Architecture

The proposed model is a Deep Recurrent Convolutional Neural Network consisting of 11 learnable layers:

  1. Convolutional Blocks: Four blocks designed to extract local acoustic patterns from the spectrogram.
  2. Recurrent Layers: Two Gated Recurrent Units (GRUs) that aggregate temporal information across the audio sequence. The inclusion of recurrence is vital, as it allows the model to maintain a "memory" of the speaker's vocal characteristics over time.
  3. Fully Connected Layer: A final layer that produces the high-level speaker embeddings.

Diarization Pipeline

The system operates in three phases. First, the R-CNN is trained in a supervised manner on a large-scale dataset to classify known speakers. Second, once the model is optimized, the final layer's activations are extracted as numerical "signatures" or embeddings. Finally, for unseen audio, the system uses cosine similarity to cluster these embeddings, effectively partitioning the audio into distinct speaker segments without requiring prior knowledge of the individuals involved.

Experimental Setup and Datasets

The model was evaluated using three established datasets and one new benchmark:

  • AMI Meeting Corpus: 100 hours of office-based interactions.
  • ICSI Meeting Corpus: 72 hours of academic discussion.
  • YouTube Speakers Corpus (YT): 550 hours of diverse, real-world audio.
  • Broadcast News Videos (BNV): A new dataset introduced by the authors containing over 6 hours of annotated content from networks like CNN and the BBC. This dataset is specifically designed to test generalization in noisy, "in-the-wild" broadcast environments.

Results and Discussion

Performance Gains

The R-CNN approach demonstrated a substantial improvement over the industry-standard LIUM baseline ( a Gaussian Mixture Model-based system). Across all tested datasets, the proposed method achieved a reduction in Diarization Error Rate (DER) of more than 30%. On the challenging BNV dataset, the R-CNN with CQT features reached a DER of 19.6%, compared to 28.5% for the baseline.

The Importance of Recurrence and CQT

A key finding of the study is the synergy between convolutional and recurrent layers. The R-CNN outperformed vanilla CNN architectures by approximately 12%, confirming that temporal context is essential for distinguishing between speakers. Furthermore, the choice of input feature proved critical; the CQT-based model consistently outperformed those using Gammatone or standard log-Mel features.

Generalization

Despite being trained on a finite set of speakers, the learned embeddings proved highly robust. The model successfully diarized audio from the BNV dataset containing entirely "unseen" speakers, suggesting that the R-CNN learns fundamental vocal characteristics rather than simply memorizing specific identities.

Conclusion

By shifting from hand-crafted features to learned embeddings via a Deep R-CNN, this research sets a new benchmark for speaker diarization. The 30% reduction in DER represents a significant step toward more reliable automated transcription and analysis. Future work will focus on improving robustness in extremely noisy environments and developing source separation techniques to better handle overlapping speech. The release of the BNV dataset further provides the research community with a rigorous tool for evaluating future diarization models.

Summary updated

Abstract

In this paper we propose a new method of speaker diarization that employs a deep learning architecture to learn speaker embeddings. In contrast to the traditional approaches that build their speaker embeddings using manually hand-crafted spectral features, we propose to train for this purpose a recurrent convolutional neural network applied directly on magnitude spectrograms. To compare our approach with the state of the art, we collect and release for the public an additional dataset of over 6 h of fully annotated broadcast material. The results of our evaluation on the new dataset and three other benchmark datasets show that our proposed method significantly outperforms the competitors and reduces diarization error rate by a large margin of over 30% with respect to the baseline.
BibTeX
@inproceedings{cyrta2017speaker,
  author = {Pawel Cyrta and Tomasz Trzcinski and Wojciech Stokowiec},
  title = {{Speaker Diarization Using Deep Recurrent Convolutional Neural Networks for Speaker Embeddings}},
  booktitle = {ISAT},
  year = {2017},
  doi = {10.1007/978-3-319-67220-5\_10},
}
Figure