Xi-Vector Embedding for Speaker Recognition

Kong Aik Lee, Qiongqiong Wang, Takafumi Koshinaka

Abstract

We present a Bayesian formulation for deep speaker embedding, wherein the xi-vector is the Bayesian counterpart of the x-vector, taking into account the uncertainty estimate. On the technology front, we offer a simple and straightforward extension to the now widely used x-vector. It consists of an auxiliary neural net predicting the frame-wise uncertainty of the input sequence. We show that the proposed extension leads to substantial improvement across all operating points, with a significant reduction in error rates and detection cost. On the theoretical front, our proposal integrates the Bayesian formulation of linear Gaussian model to speaker-embedding neural networks via the pooling layer. In one sense, our proposal integrates the Bayesian formulation of the i-vector to that of the x-vector. Hence, we refer to the embedding as the xi-vector, which is pronounced as /zai/ vector. Experimental results on the SITW evaluation set show a consistent improvement of over 17.5% in equal-error-rate and 10.9% in minimum detection cost.

Introduction

Modern automatic speaker recognition systems rely heavily on deep neural network embeddings, with the "x-vector" being the current industry standard. However, x-vectors generate rigid point estimates for speech frames, often ignoring the inherent uncertainty caused by extrinsic factors like background noise and channel distortion, as well as intrinsic factors like natural vocal tract variations.

While older generative models like the i-vector handled uncertainty using covariance matrices, they lacked the raw discriminative power of modern deep learning. This paper introduces the xi-vector (pronounced /zai/ vector), a novel deep speaker embedding that bridges this gap by integrating Bayesian uncertainty modeling directly into the standard x-vector framework.

Methodology

The xi-vector architecture modifies the standard x-vector in three key ways to account for uncertainty:

  1. Frame-wise Uncertainty Estimation: The encoder uses a two-headed neural network. One head predicts the point estimate ($z_t$), while an auxiliary head simultaneously predicts its corresponding uncertainty, represented as a diagonal precision matrix.
  2. Gaussian Posterior Inference (Temporal Pooling): Standard statistical pooling is replaced with a linear Gaussian generative model. This aggregates the sequence into a single posterior mean vector ($\phi_s$), effectively weighting each frame by its predicted certainty.
  3. Streamlined Decoder Input: Since the posterior mean vector already accounts for variability via the uncertainty weights, it is fed directly into the decoder, eliminating the need for a separate standard deviation vector.
Xi-vector architecture showing the auxiliary network for uncertainty estimation and the Gaussian posterior inference layer.

The entire network is trained end-to-end using standard stochastic gradient descent and cross-entropy loss, combining the generative strengths of the i-vector with the discriminative power of the x-vector.

Experimental Setup

The model was trained on the VoxCeleb-1 and VoxCeleb-2 datasets for wideband experiments, and Switchboard, Fisher, and MIXER corpora for narrowband experiments. Data augmentation was performed using the MUSAN dataset. Evaluation was conducted on the Speakers in the Wild (SITW) dev and eval sets, as well as the NIST SRE'18 and SRE'19 evaluation sets, which required PLDA domain adaptation.

Results and Discussion

The xi-vector demonstrates significant improvements in speaker recognition accuracy and robustness across multiple benchmarks. On the wideband SITW-Eval set, the xi-vector achieved a 17.5% relative reduction in Equal Error Rate (EER) (dropping from 2.65% to 2.19%) and a 10.9% reduction in Minimum Detection Cost Function (MinDCF) compared to the standard x-vector baseline.

Performance comparison between x-vector and xi-vector on the SITW dataset.

Key Insights

  • Uncertainty as Attention: The predicted variance acts as a dynamic weighting mechanism, allowing the network to focus on reliable frames and feature dimensions while filtering out noise.
  • Generative-Discriminative Hybrid: The paper proves that embedding a generative linear Gaussian model inside a discriminative network provides superior performance without complex multi-stage training.
  • Ablation Findings: Fine-grained, dimension-specific uncertainty modeling is crucial. Ablation studies revealed that simplifying to an isotropic (spherical) variance model led to a massive 38.7% degradation in EER.

Conclusion

The xi-vector provides a robust extension to the x-vector framework by explicitly modeling frame-level uncertainty. While the current implementation assumes diagonal covariance matrices for computational efficiency, future work could explore more complex structures to capture correlations between feature dimensions without breaking the computational budget.

Summary updated

Abstract

We present a Bayesian formulation for deep speaker embedding, wherein the xi-vector is the Bayesian counterpart of the x-vector, taking into account the uncertainty estimate. On the technology front, we offer a simple and straightforward extension to the now widely used x-vector. It consists of an auxiliary neural net predicting the frame-wise uncertainty of the input sequence. We show that the proposed extension leads to substantial improvement across all operating points, with a significant reduction in error rates and detection cost. On the theoretical front, our proposal integrates the Bayesian formulation of linear Gaussian model to speaker-embedding neural networks via the pooling layer. In one sense, our proposal integrates the Bayesian formulation of the i-vector to that of the x-vector. Hence, we refer to the embedding as the xi-vector, which is pronounced as /zai/ vector. Experimental results on the SITW evaluation set show a consistent improvement of over 17.5% in equal-error-rate and 10.9% in minimum detection cost.
BibTeX
@article{lee2021xi,
  author = {Kong Aik Lee and Qiongqiong Wang and Takafumi Koshinaka},
  title = {{Xi-Vector Embedding for Speaker Recognition}},
  journal = {IEEE Signal Processing Letters},
  year = {2021},
  url = {https://openreview.net/forum?id=OzbeDhNUOVZ},
}
Figure