Speaker Verification By Partial AUC Optimization With Mahalanobis Distance Metric Learning

Zhongxin Bai, Xiao-Lei Zhang, Jingdong Chen

Abstract

Receiver operating characteristic (ROC) and detection error tradeoff (DET) curves are standard evaluation metrics for speaker verification. However, real-world systems typically operate on specific segments of these curves depending on the application. This paper introduces a novel back-end that optimizes the Partial Area Under the ROC Curve (pAUC) using a convex Mahalanobis distance metric learning framework. By targeting specific operational "sweet spots," the proposed system achieves superior performance across multiple benchmarks, including NIST SRE16 and SITW.

What Problem Does This Paper Address?

In speaker verification (SV), the goal is to determine if two audio segments belong to the same individual. While systems are often evaluated using global metrics like the Equal Error Rate (EER) or total Area Under the Curve (AUC), these do not always align with real-world requirements.

Operational regions on the ROC curve for different applications.

As shown in the figure above, different applications prioritize different parts of the ROC curve. A bank security system requires an extremely low False Positive Rate (FPR) to prevent unauthorized access, whereas a counter-terrorism tool might prioritize a high True Positive Rate (TPR) to ensure no suspect is missed. Standard back-ends like Probabilistic Linear Discriminant Analysis (PLDA) optimize for the general data distribution, often missing these specific operational targets. This paper addresses this by allowing direct optimization of the pAUC within a user-defined range of interest $[\alpha, \beta]$.

Methods

The authors propose a back-end architecture called pAUCMetric, which integrates discriminative metric learning with traditional feature extraction.

The pAUCMetric architecture for speaker verification.

The core components of the system include:

  1. Partial AUC Optimization: The system maximizes the area under the ROC curve within a specific FPR window. This is achieved by minimizing a hinge-loss version of the pAUC objective.
  2. Mahalanobis Distance Metric Learning: Similarity between speaker embeddings (i-vectors or x-vectors) is calculated using a squared Mahalanobis distance. The learning process finds an optimal transformation matrix $M$ to maximize the pAUC.
  3. Convexity and Global Optimality: Unlike many deep learning objectives, this formulation is a convex optimization problem. This ensures that the algorithm converges to a global optimum, providing a mathematically grounded solution.
  4. Feature Preprocessing: To enhance performance, the authors utilize length-normalization and PLDA-based latent variables. This combination allows the system to benefit from both generative modeling (for feature cleaning) and discriminative metric learning (for final ranking).

Key Results

The pAUCMetric back-end was evaluated on major benchmarks, including the Cantonese and Tagalog portions of NIST SRE16 and the Speakers in the Wild (SITW) core tasks. Key findings include:

  • EER Reduction: The system achieved a relative reduction in Equal Error Rate of approximately 10% compared to standard PLDA.
  • AUC Improvement: There was a relative improvement of 20% in total AUC across evaluated tasks.
  • Language Robustness: On the Cantonese portion of NIST SRE16, pAUCMetric improved pAUC by 9% and Average Precision (AP) by 12% over the baseline x-vector systems.
  • Hard Negative Mining: By focusing on a small FPR upper bound ($\beta$), the algorithm naturally prioritizes "hard" negatives, which are critical for high-security applications.

Conclusion and Future Work

This work demonstrates that optimizing for the specific operational region of an application leads to significant performance gains in speaker verification. The use of a convex Mahalanobis metric ensures stability and optimality. Future research will explore integrating the pAUC loss directly into the end-to-end training of deep neural networks to realize these benefits at the feature-extraction level, potentially eliminating the need for separate back-end tuning.

Summary updated

Abstract

Receiver operating characteristic (ROC) and detection error tradeoff (DET) curves are two widely used evaluation metrics for speaker verification. They are equivalent since the latter can be obtained by transforming the former's true positive y-axis to false negative y-axis and then re-scaling both axes by a probit operator. Real-world speaker verification systems, however, usually work on part of the ROC curve instead of the entire ROC curve given an application. Therefore, we propose in this paper to use the area under part of the ROC curve (pAUC) as a more efficient evaluation metric for speaker verification. A Mahalanobis distance metric learning based back-end is applied to optimize pAUC, where the Mahalanobis distance metric learning guarantees that the optimization objective of the back-end is a convex one so that the global optimum solution is achievable. To improve the performance of the state-of-the-art speaker verification systems by the proposed back-end, we further propose two feature preprocessing techniques based on length-normalization and probabilistic linear discriminant analysis respectively. We evaluate the proposed systems on the major languages of NIST SRE16 and the core tasks of SITW. Experimental results show that the proposed back-end outperforms the state-of-the-art speaker verification back-ends in terms of seven evaluation metrics.
BibTeX
@article{bai2020speaker,
  author = {Zhongxin Bai and Xiao-Lei Zhang and Jingdong Chen},
  title = {{Speaker Verification By Partial AUC Optimization With Mahalanobis Distance Metric Learning}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2020},
  url = {https://openreview.net/forum?id=9clDr74S7J},
}
Figure