Optimizing a Speaker Embedding Extractor Through Backend-Driven Regularization

Luciana Ferrer, Mitchell McLarenInterspeech 2019 · OpenReview

Abstract

State-of-the-art speaker verification systems use deep neural networks (DNN) to extract highly discriminant representations of the samples, commonly called speaker embeddings. The networks are trained to maximize the cross-entropy between the estimated posteriors and the speaker labels. The pre-activations from one of the last layers in that network are used as embeddings. These sample-level vectors are then used as input to a backend that generates the final scores. The most successful backend for speaker verification to date is the probabilistic linear discriminant analysis (PLDA) backend. The full process consists of a linear discriminant analysis (LDA) projection of the embeddings, followed by mean and length normalization, ending with PLDA for score computation. While this procedure works very well compared to other approaches, it seems to be inherently suboptimal since the embeddings extractor is not directly trained to optimize the performance of the embeddings when using the PLDA backend for scoring. In this work, we propose one way to encourage the DNN to generate embeddings that are optimized for use in the PLDA backend, by adding a secondary objective designed to measure the performance of a such backend within the network. We show modest but consistent gains across several speaker recognition datasets.

Introduction

In modern speaker verification, the standard pipeline consists of two distinct components: a front-end Deep Neural Network (DNN)—typically the "x-vector" architecture—trained for speaker classification, and a backend scoring system. Once trained, an internal layer's activations are extracted as "embeddings," which are then processed via Linear Discriminant Analysis (LDA) and scored using a Probabilistic Linear Discriminant Analysis (PLDA) model.

However, a fundamental mismatch exists between these two stages. The DNN is optimized for classification accuracy (cross-entropy), while the backend relies on specific statistical assumptions about the embedding distribution. While end-to-end losses have been explored, they often struggle to outperform the robust x-vector + PLDA combination. This research proposes a middle ground: training the front-end to be inherently "PLDA-friendly" through backend-driven regularization.

Methodology

The proposed approach utilizes a dual-objective training scheme. A standard x-vector architecture is augmented with a secondary branch that mimics the evaluation pipeline.

The proposed architecture featuring primary and secondary objective branches.

The training process optimizes a hybrid loss function, $P + wS$, where:

  • Primary Objective ($P$): Standard cross-entropy loss for speaker classification.
  • Secondary Objective ($S$): A backend-driven regularizer that processes embeddings through LDA projection, mean normalization, length normalization, and a Gaussian backend (a simplified PLDA).

The weight $w$ is scheduled to increase during training, allowing the model to first establish basic features before refining the embedding space according to backend constraints. Importantly, the parameters for the secondary branch (LDA matrix and Gaussian statistics) are re-estimated periodically using a subset of training data, ensuring the simulated backend adapts as the embeddings evolve.

Experimental Setup

The model was trained on a comprehensive collection of datasets, including NIST SRE (2004-2012), Mixer6, and VoxCeleb 1 & 2, comprising over 234,000 signals from nearly 15,000 speakers. Robustness was evaluated across a diverse range of benchmarks:

  • SITW (Speakers in the Wild): Real-world media.
  • SRE16 & SRE18: NIST evaluations focusing on telephony and VoIP.
  • LASRS: Bilingual corpora (Arabic, Korean, Spanish).
  • RATS: Noisy radio traffic.
  • FVC Aus: Forensic data in Australian English.

Results and Discussion

The proposed regularization method yielded modest but consistent performance improvements across nearly all test sets. The primary classification loss and the secondary backend loss were found to be complementary: the primary loss ensures discriminative power, while the secondary loss structures the embedding space for optimal backend interpretation.

Performance comparison (EER and DCF) between the baseline and regularized models across various datasets.

Significant gains were observed in the SRE16 Evaluation, with a 13.4% relative improvement in Equal Error Rate (EER) and a 16.5% improvement in Detection Cost Function (DCF). The SRE18 Evaluation showed an 8.3% relative improvement in EER. These results demonstrate that the regularization helps the model generalize better to unseen domains, including noisy conditions (RATS) and different languages (LASRS).

A slight degradation was noted on the exceptionally clean FVC Aus dataset, suggesting that this form of structural regularization is most beneficial for handling the variability and noise present in "in-the-wild" data.

Conclusion

By integrating backend assumptions directly into the training objective, we can bridge the gap between embedding extraction and scoring. This backend-driven regularization encourages the DNN to produce embeddings that are better suited for PLDA-based verification. Future work may explore more complex PLDA variants within the secondary branch or the application of this technique to alternative DNN architectures.

Summary updated

Abstract

State-of-the-art speaker verification systems use deep neural networks (DNN) to extract highly discriminant representations of the samples, commonly called speaker embeddings. The networks are trained to maximize the cross-entropy between the estimated posteriors and the speaker labels. The pre-activations from one of the last layers in that network are used as embeddings. These sample-level vectors are then used as input to a backend that generates the final scores. The most successful backend for speaker verification to date is the probabilistic linear discriminant analysis (PLDA) backend. The full process consists of a linear discriminant analysis (LDA) projection of the embeddings, followed by mean and length normalization, ending with PLDA for score computation. While this procedure works very well compared to other approaches, it seems to be inherently suboptimal since the embeddings extractor is not directly trained to optimize the performance of the embeddings when using the PLDA backend for scoring. In this work, we propose one way to encourage the DNN to generate embeddings that are optimized for use in the PLDA backend, by adding a secondary objective designed to measure the performance of a such backend within the network. We show modest but consistent gains across several speaker recognition datasets.
BibTeX
@inproceedings{ferrer2019optimizing,
  author = {Luciana Ferrer and Mitchell McLaren},
  title = {{Optimizing a Speaker Embedding Extractor Through Backend-Driven Regularization}},
  booktitle = {Interspeech},
  year = {2019},
  url = {https://openreview.net/forum?id=IbrAw_c_ZDQ},
}

Loading related content...

Figure