Abstract
In modern speaker verification (SV) systems, the choice of back-end scoring—the method used to compare two speaker embeddings—is critical. While Probabilistic Linear Discriminant Analysis (PLDA) was long considered the gold standard, recent deep learning-based embeddings often perform better with simple Cosine similarity. This paper resolves this paradox by mathematically demonstrating that Cosine scoring is a special, restricted case of PLDA. Through experimental analysis, we show that the superior performance of Cosine scoring in domain-matched scenarios is primarily due to its inherent assumption of dimensional independence, which acts as a form of regularization. Conversely, in domain-mismatched scenarios, the flexibility of full-covariance PLDA remains superior for adaptation.
Introduction
Speaker verification involves determining whether two speech segments belong to the same individual. This process typically follows a two-stage pipeline: a front-end neural network extracts high-dimensional embeddings (such as x-vectors), and a back-end model computes a similarity score.
Historically, PLDA was the preferred back-end because it explicitly models within-speaker and between-speaker variability. However, with the advent of neural embeddings trained using angular margin losses (like AAM-softmax), researchers observed a counter-intuitive trend: the "simpler" Cosine similarity often outperformed the "sophisticated" PLDA. This study investigates the mathematical relationship between these two approaches to explain why and when each method excels.
Theoretical Unification
The core contribution of this work is the mathematical proof that Cosine similarity is not a distinct alternative to PLDA, but rather a specific configuration of it. By analyzing a two-covariance PLDA model, the authors show that if the within-class and between-class covariance matrices are constrained to identity matrices, the resulting log-likelihood ratio simplifies exactly to the Cosine similarity score.
This unification reveals that Cosine scoring implicitly adopts all the Gaussian assumptions of PLDA, while adding a much stricter constraint: it assumes that all dimensions of the embedding are mutually independent and share the same variance.
Methodology
To isolate the impact of these assumptions, the researchers introduced Diagonal PLDA (DPLDA). Unlike standard PLDA, which uses full covariance matrices to model correlations between embedding dimensions, DPLDA forces these matrices to be diagonal. This creates a middle ground that retains the independence assumption of Cosine scoring while allowing the model to learn variances from speaker labels.
Experiments were conducted across two distinct conditions:
- Domain-Matched: Using the VoxCeleb1 & 2 datasets, where the training and testing data consist of similar celebrity interview audio.
- Domain-Mismatched: Using CNCeleb1, where a model trained on English interviews (VoxCeleb) is tested on Chinese speech across various acoustic environments.
Key Results and Analysis
The Matched Domain Paradox
In the VoxCeleb experiments, the performance gap was stark. Cosine similarity achieved an Equal Error Rate (EER) of 1.06%, while standard PLDA lagged significantly at 1.86%. However, when the researchers applied DPLDA, the EER dropped to 1.11%, nearly matching the Cosine baseline.
This result proves that the "secret sauce" of Cosine scoring in matched domains is the dimensional independence assumption. Modern neural networks, particularly those trained with margin-based losses, are highly effective at decorrelating features. By "hard-coding" this independence, Cosine scoring prevents the back-end from over-fitting to spurious correlations in the training data, providing a robust regularization effect.
The Mismatched Domain Shift
The situation reverses when the system encounters a domain shift. On the CNCeleb dataset, standard PLDA outperformed both Cosine and DPLDA:
- Standard PLDA: 8.90% EER
- Cosine: 10.11% EER
- DPLDA: 10.24% EER
In mismatched conditions, the embedding dimensions often become correlated in ways the front-end model did not anticipate. In these cases, the restrictive independence assumption of Cosine scoring becomes a liability. The full-covariance PLDA model, despite its complexity, is flexible enough to capture and compensate for these new correlations, making it the superior choice for domain adaptation.
Conclusion
This research clarifies the relationship between the two most common back-ends in speaker verification. We conclude that Cosine scoring is essentially a "regularized PLDA" that excels when the front-end has already successfully decorrelated the embedding space. However, the flexibility of the full PLDA framework remains essential for handling domain mismatch. Future work should focus on hybrid models that can adaptively tune the degree of dimensional independence based on the characteristics of the target domain.
Summary updated