Unifying Cosine and PLDA Back-ends for Speaker Verification

Zhiyuan Peng, Xuanji He, Ke Ding, Tan Lee, Guanglu Wan

Abstract

In modern speaker verification (SV) systems, the choice of back-end scoring—the method used to compare two speaker embeddings—is critical. While Probabilistic Linear Discriminant Analysis (PLDA) was long considered the gold standard, recent deep learning-based embeddings often perform better with simple Cosine similarity. This paper resolves this paradox by mathematically demonstrating that Cosine scoring is a special, restricted case of PLDA. Through experimental analysis, we show that the superior performance of Cosine scoring in domain-matched scenarios is primarily due to its inherent assumption of dimensional independence, which acts as a form of regularization. Conversely, in domain-mismatched scenarios, the flexibility of full-covariance PLDA remains superior for adaptation.

Introduction

Speaker verification involves determining whether two speech segments belong to the same individual. This process typically follows a two-stage pipeline: a front-end neural network extracts high-dimensional embeddings (such as x-vectors), and a back-end model computes a similarity score.

Historically, PLDA was the preferred back-end because it explicitly models within-speaker and between-speaker variability. However, with the advent of neural embeddings trained using angular margin losses (like AAM-softmax), researchers observed a counter-intuitive trend: the "simpler" Cosine similarity often outperformed the "sophisticated" PLDA. This study investigates the mathematical relationship between these two approaches to explain why and when each method excels.

Theoretical Unification

The core contribution of this work is the mathematical proof that Cosine similarity is not a distinct alternative to PLDA, but rather a specific configuration of it. By analyzing a two-covariance PLDA model, the authors show that if the within-class and between-class covariance matrices are constrained to identity matrices, the resulting log-likelihood ratio simplifies exactly to the Cosine similarity score.

This unification reveals that Cosine scoring implicitly adopts all the Gaussian assumptions of PLDA, while adding a much stricter constraint: it assumes that all dimensions of the embedding are mutually independent and share the same variance.

Methodology

To isolate the impact of these assumptions, the researchers introduced Diagonal PLDA (DPLDA). Unlike standard PLDA, which uses full covariance matrices to model correlations between embedding dimensions, DPLDA forces these matrices to be diagonal. This creates a middle ground that retains the independence assumption of Cosine scoring while allowing the model to learn variances from speaker labels.

Experiments were conducted across two distinct conditions:

  1. Domain-Matched: Using the VoxCeleb1 & 2 datasets, where the training and testing data consist of similar celebrity interview audio.
  2. Domain-Mismatched: Using CNCeleb1, where a model trained on English interviews (VoxCeleb) is tested on Chinese speech across various acoustic environments.

Key Results and Analysis

The Matched Domain Paradox

In the VoxCeleb experiments, the performance gap was stark. Cosine similarity achieved an Equal Error Rate (EER) of 1.06%, while standard PLDA lagged significantly at 1.86%. However, when the researchers applied DPLDA, the EER dropped to 1.11%, nearly matching the Cosine baseline.

This result proves that the "secret sauce" of Cosine scoring in matched domains is the dimensional independence assumption. Modern neural networks, particularly those trained with margin-based losses, are highly effective at decorrelating features. By "hard-coding" this independence, Cosine scoring prevents the back-end from over-fitting to spurious correlations in the training data, providing a robust regularization effect.

The Mismatched Domain Shift

The situation reverses when the system encounters a domain shift. On the CNCeleb dataset, standard PLDA outperformed both Cosine and DPLDA:

  • Standard PLDA: 8.90% EER
  • Cosine: 10.11% EER
  • DPLDA: 10.24% EER

In mismatched conditions, the embedding dimensions often become correlated in ways the front-end model did not anticipate. In these cases, the restrictive independence assumption of Cosine scoring becomes a liability. The full-covariance PLDA model, despite its complexity, is flexible enough to capture and compensate for these new correlations, making it the superior choice for domain adaptation.

Conclusion

This research clarifies the relationship between the two most common back-ends in speaker verification. We conclude that Cosine scoring is essentially a "regularized PLDA" that excels when the front-end has already successfully decorrelated the embedding space. However, the flexibility of the full PLDA framework remains essential for handling domain mismatch. Future work should focus on hybrid models that can adaptively tune the degree of dimensional independence based on the characteristics of the target domain.

Summary updated

Abstract

State-of-art speaker verification (SV) systems use a back-end model to score the similarity of speaker embeddings extracted from a neural network model. The commonly used back-end models are the cosine scoring and the probabilistic linear discriminant analysis (PLDA) scoring. With the recently developed neural embeddings, the theoretically more appealing PLDA approach is found to have no advantage against or even be inferior the simple cosine scoring in terms of SV system performance. This paper presents an investigation on the relation between the two scoring approaches, aiming to explain the above counter-intuitive observation. It is shown that the cosine scoring is essentially a special case of PLDA scoring. In other words, by properly setting the parameters of PLDA, the two back-ends become equivalent. As a consequence, the cosine scoring not only inherits the basic assumptions for the PLDA but also introduces additional assumptions on the properties of input embeddings. Experiments show that the dimensional independence assumption required by the cosine scoring contributes most to the performance gap between the two methods under the domain-matched condition. When there is severe domain mismatch and the dimensional independence assumption does not hold, the PLDA would perform better than the cosine for domain adaptation.
BibTeX
@inproceedings{peng2022unifying,
  author = {Zhiyuan Peng and Xuanji He and Ke Ding and Tan Lee and Guanglu Wan},
  title = {{Unifying Cosine and PLDA Back-ends for Speaker Verification}},
  booktitle = {Interspeech},
  year = {2022},
  url = {https://openreview.net/forum?id=23tNjgoTOp},
}
Figure