Online Speaker Diarization with Relation Network

Xiang Li, Yucheng Zhao, Chong Luo, Wenjun ZengarXiv 2020 · OpenReview

TL;DR

This paper introduces RenoSD, an end-to-end online speaker diarization system that integrates voice activity detection (VAD), speaker embedding, and speaker identity association into a single neural network. By leveraging a Relation Network for meta-learning speaker identity association, RenoSD achieves state-of-the-art performance with lower complexity and less training data compared to existing methods.

What Problem Does This Paper Address?

Speaker diarization, the task of identifying "who spoke when" in an audio recording, is crucial for many applications like meeting transcription and information retrieval. Traditional diarization systems often rely on a pipeline of independently optimized modules: voice activity detection (VAD) to find speech segments, speaker embedding extraction to represent speech characteristics, and clustering to group segments by speaker. This modular approach can lead to error propagation and suboptimal performance. Furthermore, existing systems are often designed for offline processing, requiring the entire audio to be available beforehand. This is a significant limitation for real-time applications like live captioning, where decisions must be made as the audio streams in. The challenge is to develop an efficient, accurate, and truly online diarization system that can handle these demands.

Methods

The core innovation of RenoSD is its end-to-end architecture built around a Relation Network (RN). Unlike traditional methods that use fixed distance metrics for speaker comparison, RenoSD's RN learns a deep, data-driven distance metric. This allows it to directly output a "relation score" (0-1) indicating the probability that two audio segments belong to the same speaker.

Key methodological aspects include:

  • Integrated Modules: VAD, embedding extraction, and speaker identity association are unified into a single neural network. This allows for joint optimization, where each component benefits from the others.
  • Relation Network for Speaker Association: The RN takes pairs of speaker embeddings and learns to predict their similarity. This meta-learning approach is powerful for few-shot learning scenarios and robust speaker discrimination.
  • Online Processing: The system processes audio in short, non-overlapping segments. For a new segment, its embedding is compared against the earliest embeddings of currently registered speakers. This avoids the need for global context and enables real-time operation.
  • VAD Integration: A speech/non-speech classifier is integrated into the network, allowing for joint training and improving VAD performance alongside diarization.

Datasets

  • VoxCeleb1&2: Used for pre-training the system. This is a large-scale dataset of celebrity speech.
  • AMI Corpus: Used for fine-tuning and evaluation. This corpus contains recordings of group meetings.
  • CALLHOME Dataset: Used for evaluation. This is a well-known benchmark for speaker diarization, containing diverse audio in multiple languages.

Key Insights and Ideas

  • End-to-End Integration is Key: Unifying VAD, embedding, and speaker association into a single network significantly improves performance. The modules mutually benefit from joint training, leading to better overall accuracy.
  • Meta-Learning for Speaker Identity: The Relation Network's ability to learn a flexible, data-driven similarity metric is more effective than fixed distance measures for distinguishing speakers, especially in challenging conditions.
  • Online Efficiency: The strategy of comparing new segments only with the "earliest" representative of each existing speaker queue is a clever way to maintain low computational complexity in an online setting.
  • VAD as a Joint Task: Integrating VAD into the diarization network not only improves the VAD component but also positively impacts the speaker association, demonstrating the synergy of joint training.

Key Results

  • Improved DER on AMI: RenoSD achieved a Diarization Error Rate (DER) of 15.77% on the AMI corpus, outperforming the state-of-the-art x-vector baseline (which achieved 19.75% even with oracle VAD and speaker count information).
  • Superior Performance on CALLHOME: On the CALLHOME dataset, RenoSD achieved a DER of 10.13%, surpassing the best version of the online UIS-RNN system (7.6%) and even outperforming x-vector with oracle speaker counts (10.90%).
  • Reduced Training Data and Complexity: Compared to UIS-RNN, RenoSD achieved better performance using significantly less training data and exhibiting lower time complexity, which scales with the number of speakers rather than the number of segments.
  • Effective VAD Integration: The integrated VAD module significantly reduced False Alarm (FA) and Miss Error (MI) rates on the AMI corpus, demonstrating its effectiveness.

Limitations and Future Work

  • Computational Complexity: While RenoSD's complexity is bounded by the number of speakers, it can still become a bottleneck if the number of speakers is very large. The paper notes that processing speed is faster when the number of speakers is less than the number of segments.
  • Future Work: The paper doesn't explicitly detail future work, but the success of the end-to-end, relation-network-based approach suggests potential for further exploration in other sequence-to-sequence tasks or in adapting this architecture to different audio processing challenges. The ability to learn a flexible similarity metric could be beneficial in scenarios with varying speaker characteristics or noisy environments.

Summary updated

Abstract

In this paper, we propose an online speaker diarization system based on Relation Network, named RenoSD. Unlike conventional diariztion systems which consist of several independently-optimized modules, RenoSD implements voice-activity-detection (VAD), embedding extraction, and speaker identity association using a single deep neural network. The most striking feature of RenoSD is that it adopts a meta-learning strategy for speaker identity association. In particular, the relation network learns to learn a deep distance metric in a data-driven way and it can determine through a simple forward pass whether two given segments belong to the same speaker. As such, RenoSD can be performed in an online manner with low latency. Experimental results on AMI and CALLHOME datasets show that the proposed RenoSD system achieves consistent improvements over the state-of-the-art x-vector baseline. Compared with an existing online diarization system named UIS-RNN, RenoSD achieves a better performance using much fewer training data and at a lower time complexity.
BibTeX
@misc{li2020online,
  author = {Xiang Li and Yucheng Zhao and Chong Luo and Wenjun Zeng},
  title = {{Online Speaker Diarization with Relation Network}},
  year = {2020},
  eprint = {2009.08162},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.08162},
}
Figure