Centroid-based Deep Metric Learning for Speaker Recognition

Jixuan Wang, Kuan-Chieh Wang, Marc T. Law, Frank Rudzicz, Michael Brudno

Abstract

Speaker recognition systems often struggle with a "generalization gap" when encountering speakers not seen during training. This paper explores the application of Prototypical Network Loss (PNL)—originally developed for few-shot image classification—to the domain of speaker recognition. By representing speakers as centroids in an embedding space, PNL achieves superior performance on unseen speakers and significantly faster training times compared to the traditional Triplet Loss.

Introduction

Modern speaker verification and identification systems face significant challenges in few-shot learning scenarios, where a system must recognize an individual based on a very limited amount of enrollment audio. The industry standard, Triplet Loss (TL), suffers from high sampling complexity, sensitivity to hyperparameters like the margin, and a local focus that fails to enforce global cluster cohesion. This research addresses these limitations by proposing a centroid-based approach.

Methodology

The authors utilize Prototypical Networks to learn robust speaker embeddings. Unlike Triplet Loss, which compares an anchor to individual positive and negative samples, PNL operates on "episodes" consisting of a support set and a query set.

Prototypical Network Loss

For each speaker in a mini-batch, a prototype (or centroid) is computed as the mean embedding of several utterances from the support set. Query utterances are then compared against all prototypes in the batch. The model is trained to minimize the distance between a query and its corresponding prototype using a softmax over distances. This approach acts as a form of supervised clustering, encouraging the model to group all utterances of a speaker into a single, tight cluster.

Comparison between (a) Prototypical Network Loss and (b) Triplet Loss.

Model Architecture

To isolate the impact of the loss function, a consistent backbone was used:

  • Input: 59-dimensional acoustic features (MFCCs + derivatives).
  • Encoder: A bidirectional LSTM (BiLSTM).
  • Aggregation: Outputs are average-pooled, passed through a fully connected layer, and normalized to a 16-dimensional embedding.

Experimental Setup

Evaluations were conducted on two datasets:

  1. VCTK: Clean, studio-quality speech from 108 speakers.
  2. VoxCeleb2: Challenging "in-the-wild" audio with background noise. In both cases, a subset of speakers was reserved as an "unseen" set to evaluate few-shot generalization.

Key Results

PNL demonstrated significant improvements over various Triplet Loss configurations. While performance was comparable on "seen" speakers, the advantage of PNL became starkly evident on the "unseen" set.

ROC curves and AUC for Test and Unseen sets across different loss functions.
  • Speaker Verification: On unseen VCTK speakers, PNL achieved an Equal Error Rate (EER) of 12.00%, compared to 13.44% for the best TL variant.
  • Speaker Identification: In an 18-way task for unseen speakers, 3-shot PNL reached 69.64% accuracy, a 19% relative improvement over the best TL model (58.51%).
  • Efficiency: PNL trained approximately 3x faster than TL by avoiding the combinatorial explosion of training pairs and the need for complex triplet mining.

Discussion and Conclusion

The results suggest that comparing samples to a class-level "mean" representation is more robust than comparing to single, potentially noisy anchors. PNL also simplifies the training process by removing the need for sensitive margin hyperparameters. Future work will explore scaling these findings to higher embedding dimensions and integrating PNL with advanced architectures like ResNets for applications such as speaker diarization.

Summary updated

BibTeX
@inproceedings{wang2019centroid,
  author = {Jixuan Wang and Kuan-Chieh Wang and Marc T. Law and Frank Rudzicz and Michael Brudno},
  title = {{Centroid-based Deep Metric Learning for Speaker Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=RDIFlj-cSoC},
}
Figure