Neural PLDA Modeling for End-to-End Speaker Verification

Shreyas Ramoji, Prashant Krishnan V, Sriram Ganapathy

Abstract

Traditional speaker verification systems rely on a modular approach, separating embedding extraction from similarity scoring. This paper proposes an end-to-end (E2E) architecture that jointly optimizes a neural embedding extractor and a discriminative Neural PLDA (NPLDA) backend. By utilizing a differentiable approximation of the Minimum Detection Cost Function (minDCF), the system directly minimizes the evaluation metric during training. Experimental results on NIST SRE benchmarks demonstrate relative improvements of up to 33% over standard modular baselines.

Introduction

Automatic Speaker Verification (ASV) typically involves two distinct stages: a front-end deep neural network for extracting fixed-length speaker embeddings (x-vectors) and a back-end model, such as Probabilistic Linear Discriminant Analysis (PLDA), for scoring. However, this modularity creates a mismatch of objectives. The front-end is often optimized for closed-set classification, while the back-end handles open-set verification. Furthermore, standard PLDA is a generative model that may lack the robustness of discriminative neural approaches. This work addresses these issues by unifying both stages into a single, differentiable neural network optimized directly for the verification task.

Proposed Methodology

The core of the proposed system is the joint optimization of the embedding extractor and the scoring backend.

Architecture

The model processes Mel-frequency cepstral coefficients (MFCCs) from enrollment and test utterances through a shared 10-layer Extended TDNN. The resulting frame-level features are aggregated via a statistics pooling layer.

The End-to-End x-vector NPLDA architecture, showing the flow from MFCC input to final verification score.

Neural PLDA (NPLDA) and Differentiable Loss

The traditional PLDA scoring logic is reformulated as a series of neural network layers, including affine transformations, unit-length normalization, and a quadratic scoring layer. This makes the entire pipeline differentiable. To train the system, the authors introduce a "soft" version of the Normalized Detection Cost Function (DCF). By using a sigmoid-based approximation of discrete counts (misses and false alarms), the network can minimize the actual evaluation metric via backpropagation.

Training and Sampling

Due to the memory intensity of E2E training on raw audio features, a specialized sampling strategy was employed. Batches were constructed with 64 utterances to generate 1024 trials, maintaining a balance between target and non-target pairs. The model is initialized using pre-trained modular components to ensure stability during joint optimization.

Experimental Results

The system was evaluated on the NIST SRE 2018 Development set and the NIST SRE 2019 Evaluation set, using models trained on large-scale datasets including VoxCeleb and previous NIST SRE collections.

Performance comparison of baseline GPLDA, NPLDA, and E2E models across different pooling functions and benchmarks.

As shown in the results table, the E2E system significantly outperformed the state-of-the-art Kaldi-based Gaussian PLDA (GPLDA) baseline. On the SRE18 Dev set, the E2E model with variance pooling achieved a minDCF of 0.307, compared to 0.436 for the baseline. On the SRE19 Eval set, the E2E system reached a minDCF of 0.405. Overall, the E2E approach yielded relative improvements ranging from 20% to 33% in minDCF across evaluation sets.

Conclusion and Future Work

The integration of embedding extraction and scoring into a single differentiable framework, combined with a metric-linked loss function, provides a powerful paradigm for speaker verification. While the current system relies on modular initialization and faces significant GPU memory constraints, it sets a new performance benchmark. Future research will explore more complex front-end architectures, such as LSTMs or attention mechanisms, and advanced sampling techniques like curriculum learning to further enhance robustness.

Summary updated

BibTeX
@inproceedings{ramoji2020neural,
  author = {Shreyas Ramoji and Prashant Krishnan V and Sriram Ganapathy},
  title = {{Neural PLDA Modeling for End-to-End Speaker Verification}},
  booktitle = {Interspeech},
  year = {2020},
  doi = {10.21437/interspeech.2020-2699},
}
Figure