Speaker recognition based on deep learning: An overview

Zhongxin Bai, Xiao-Lei Zhang

TL;DR

This paper provides a comprehensive overview of the rapidly evolving field of speaker recognition, specifically focusing on how deep learning has revolutionized the task. It surveys the latest advancements in speaker verification, identification, diarization, and robust recognition, highlighting deep learning's ability to extract highly abstract and effective speaker embeddings.

What Problem Does This Paper Address?

Speaker recognition, the task of identifying individuals by their voice, has seen dramatic improvements with the advent of deep learning. However, the field is advancing so quickly that a consolidated understanding of the latest techniques, especially those leveraging deep learning, is lacking. This paper aims to bridge that gap by providing a structured overview of the state-of-the-art in deep learning-based speaker recognition, covering its various subtasks and challenges.

Methods

The paper delves into several key deep learning methodologies for speaker recognition:

  • Deep Speaker Feature Extraction: This is a core focus, exploring how deep neural networks (DNNs) can learn powerful representations (embeddings) from audio. This includes examining different network architectures (DNNs, CNNs, RNNs/LSTMs), input features (raw audio, spectrograms, MFCCs), and temporal pooling strategies (average, statistics, attention-based) to aggregate frame-level information into utterance-level embeddings.
  • DNN-UBM/i-vector and DNN-BNF/i-vector: These are hybrid approaches that leverage DNNs to improve traditional i-vector systems. DNN-UBM uses DNNs to generate posteriors for i-vector extraction, while DNN-BNF extracts features from a bottleneck layer of a DNN.
  • Deep Embeddings (d-vector, x-vector): These methods directly learn speaker embeddings using DNNs trained in a supervised manner, often outperforming traditional i-vector systems.
  • End-to-End Speaker Verification: This approach directly learns a similarity score between utterance pairs using DNNs, bypassing traditional multi-stage pipelines. Various loss functions (pairwise, triplet, quadruplet) are discussed for this task.
  • Speaker Diarization: The paper covers stage-wise methods (segmentation, feature extraction, clustering) and emerging end-to-end, online, and multimodal approaches.
  • Robust Speaker Recognition: Techniques like domain adaptation (adversarial, reconstruction, discrepancy-based) and speech enhancement are reviewed to handle noisy and mismatched conditions.

Datasets

The paper references several standard datasets commonly used in speaker recognition research:

  • NIST SRE: A long-standing benchmark for speaker recognition evaluations.
  • VoxCeleb1, 2: Large-scale datasets collected from YouTube, featuring diverse speakers and conditions.
  • SITW (Speakers in the Wild): Designed for benchmarking text-independent speaker recognition in unconstrained environments.
  • RSR2015: Recorded in typical office environments using mobile devices.
  • RedDots: Collected using a mobile app with online recording and upload.
  • VOICES: Features speech in complex environments with background noise and overlapping speech.
  • LibriSpeech: Primarily for ASR but useful for speaker recognition due to speaker identity labels.
  • CN-CELEB: A Chinese speaker recognition dataset.
  • BookTubeSpeech: Collected via an automatic pipeline from BookTube videos.
  • Hi-MIA: Designed for far-field scenarios with microphone arrays.
  • FFSVC 2020: A far-field speaker verification challenge focusing on noisy, distributed microphone arrays.
  • DIHARD1, 2: Challenges focused on improving speaker diarization robustness to variations in recording equipment, noise, and conversational domains.
  • AMI: A multimodal dataset for meeting recordings.

Key Insights and Ideas

  • Deep Learning's Representation Power: The core advantage of deep learning lies in its ability to learn highly abstract and discriminative speaker embeddings, significantly outperforming traditional handcrafted features.
  • Shift from Generative to Discriminative Models: The field has largely moved from generative models (like GMM-UBM) to discriminative models (DNN-based embeddings) for better performance.
  • End-to-End Systems are Gaining Traction: While stage-wise systems are still prevalent, end-to-end approaches for verification and diarization are showing promise, simplifying pipelines and potentially improving performance.
  • Loss Functions are Crucial: The choice of objective function (e.g., Softmax variants, triplet loss, pairwise loss) plays a critical role in learning effective speaker embeddings and achieving high performance.
  • Robustness is Key for Real-World Deployment: Addressing domain mismatch and noise through techniques like domain adaptation and speech enhancement is essential for practical speaker recognition systems.
  • X-vectors are a Dominant Architecture: The x-vector, a segment-level embedding derived from TDNNs, has become a highly influential architecture in deep speaker recognition.

Key Results

The paper highlights significant performance improvements achieved by deep learning methods:

  • EER Reductions: DNN-UBM/i-vector systems showed up to 30% relative Equal Error Rate (EER) reduction compared to GMM/i-vector on telephone conditions.
  • X-vector Superiority: X-vectors have achieved state-of-the-art performance, significantly outperforming GMM/i-vector baselines, especially with data augmentation.
  • Robustness Gains: Deep learning methods, particularly with domain adaptation and speech enhancement, have demonstrated substantial improvements in handling noisy and mismatched conditions, enabling "recognition in the wild."
  • Diarization Accuracy: Deep learning has also advanced speaker diarization, leading to more accurate segmentation and identification of speakers in conversations.

Limitations and Future Work

  • Need for Labeled Data: Many deep learning methods, especially for feature extraction and supervised training, still require large amounts of labeled data, which can be costly and time-consuming to acquire.
  • Computational Complexity: Deep models often have a large number of parameters, leading to high computational costs for training and inference, posing challenges for deployment on resource-constrained devices.
  • Theoretical Understanding of Loss Functions: While many effective loss functions have been proposed, a strong theoretical foundation for their success and guidance for designing new ones is still lacking.
  • Real-World Diarization Challenges: Speaker diarization in real-world scenarios remains difficult due to issues like unknown speaker counts, rapid speaker changes, speech overlap, and background noise.
  • Domain Adaptation Progress: While adversarial learning has shown promise, its progress in speaker recognition has not yet surpassed traditional methods like PLDA-based adaptation.
  • Raw Waveform Networks: The paper notes that raw waveform networks, which could potentially capture more information than handcrafted features, have not yet attracted widespread attention.
  • Model Size and Inference Efficiency: Further research is needed to optimize model size and inference efficiency for edge and mobile devices.

Summary updated

Abstract

Speaker recognition is a task of identifying persons from their voices. Recently, deep learning has dramatically revolutionized speaker recognition. However, there is lack of comprehensive reviews on the exciting progress. In this paper, we review several major subtasks of speaker recognition, including speaker verification, identification, diarization, and robust speaker recognition, with a focus on deep-learning-based methods. Because the major advantage of deep learning over conventional methods is its representation ability, which is able to produce highly abstract embedding features from utterances, we first pay close attention to deep-learning-based speaker feature extraction, including the inputs, network structures, temporal pooling strategies, and objective functions respectively, which are the fundamental components of many speaker recognition subtasks. Then, we make an overview of speaker diarization, with an emphasis of recent supervised, end-to-end, and online diarization. Finally, we survey robust speaker recognition from the perspectives of domain adaptation and speech enhancement, which are two major approaches of dealing with domain mismatch and noise problems. Popular and recently released corpora are listed at the end of the paper.
BibTeX
@inproceedings{bai2021speaker,
  author = {Zhongxin Bai and Xiao-Lei Zhang},
  title = {{Speaker recognition based on deep learning: An overview}},
  booktitle = {Neural Networks},
  year = {2021},
  url = {https://openreview.net/forum?id=4EbBnKG0iy},
}

Loading related content...

Figure