Improving Fairness in Speaker Verification via Group-Adapted Fusion Network

Hua Shen, Yuguang Yang, Guoli Sun, Ryan Langman, Eunjung Han, Jasha Droppo, Andreas Stolcke

TL;DR

This paper tackles the problem of fairness in speaker verification, where models often perform worse for underrepresented demographic groups due to biased training data. They propose a novel "Group-Adapted Fusion Network" (GFN) that uses specialized encoders for different groups and fuses their outputs to achieve more equitable performance across all speakers.

What Problem Does This Paper Address?

Speaker verification systems, used in everything from smart assistants to secure access, are trained on large datasets to distinguish between speakers. However, these datasets often have imbalanced representation across demographic groups (e.g., gender, age, accent). This imbalance can lead to models learning voice characteristics that are more prominent in dominant groups, causing them to perform poorly for underrepresented groups. This "fairness gap" means that certain users might not receive the same level of service reliability as others, which is a significant issue for equitable access to technology. Current performance metrics, like overall Equal Error Rate (EER), can mask these disparities.

Methods

The core of the proposed solution is the Group-Adapted Fusion Network (GFN), a modular architecture with two main components:

  1. Group Embedding Adaptation (Front-End): Instead of a single "base" encoder, GFN uses multiple encoders. A base encoder is first trained on mixed-gender data to capture general voice characteristics. Then, this base encoder is fine-tuned separately for specific demographic groups (in this paper, male and female). This "group-adapted" encoder learns to extract more discriminative features tailored to the specific voice characteristics of that group.
  2. Score Fusion (Back-End): The embeddings from the base encoder and the group-adapted encoders are used to compute similarity scores. These scores are then fed into a learnable score fusion model (a simple Multi-Layer Perceptron or MLP). This fusion mechanism allows the system to leverage the strengths of each specialized encoder and prevent overfitting to any single group.

Datasets

  • Training: The authors created custom training datasets from the VoxCeleb2 dataset, referred to as VoxCeleb2-GRC (Gender Ratio Controlled). These datasets were designed to have varying degrees of gender imbalance, with female-to-male (F:M) ratios ranging from 9:1 to 1:9, while keeping the total number of speakers constant.
  • Evaluation: For evaluation, they used a specially constructed dataset called VoxCeleb1-F (Fairness), derived from VoxCeleb1. This dataset was carefully designed to control for the types of trials (positive/negative, same/different gender) to rigorously assess fairness metrics.

Key Insights and Ideas

  • Imbalance breeds unfairness: The paper provides direct evidence that imbalanced training data leads to a performance disparity, with minority groups suffering from higher error rates. This is not just a theoretical concern but a practical one.
  • Specialization matters: Adapting encoders to specific demographic groups allows them to learn more nuanced features, improving their ability to represent those groups accurately.
  • Fusion is key: Simply having specialized encoders isn't enough. Fusing their outputs is crucial to combine their strengths and achieve better overall performance and fairness, preventing one specialized encoder from dominating or causing other issues.
  • Overall EER can be misleading: The study highlights that improvements in overall EER don't necessarily translate to fairness; the performance gap between groups can persist or even widen.

Key Results

The GFN model demonstrated significant improvements in fairness:

  • Overall EER Reduction: Relative reduction of 9.6% to 29.0% compared to baselines.
  • Minority Group EER Reduction: Reduced minority group EER by 13.7% to 18.6%.
  • EER Disparity Reduction: Decreased the EER disparity between groups by 20.0% to 25.4%.
  • Improved Group-wise Performance: The GFN consistently achieved better group-wise EERs (e.g., female group EER of 3.12% and male group EER of 5.88% in a 9:1 F:M imbalance scenario) compared to baseline models.
  • Compact and Separated Embeddings: Visualization of embeddings using t-SNE showed that GFN produces more compact speaker clusters with better separation, indicating improved representation quality.

Limitations and Future Work

  • Generalizability to other groups: While the approach is proposed to be applicable to other demographic groups (children, elderly, accents), the current evaluation is limited to gender.
  • Overfitting risk: The authors acknowledge that performing group adaptation and fusion on the training dataset might introduce additional overfitting risk, especially when applying a trained GFN to out-of-domain datasets.
  • Backend scoring: They suggest that robust backend scoring approaches like PLDA could be explored further to alleviate model unfairness.
  • Alternative adaptation methods: The paper explored and compared different adaptation strategies, finding that separate encoders and score fusion were superior to methods like "gender batching with weighted loss" (GBWL).

Summary updated

Abstract

Modern speaker verification models use deep neural networks to encode utterance audio into discriminative embedding vectors. During the training process, these networks are typically optimized to differentiate arbitrary speakers. This learning process biases the learning of fine voice characteristics towards dominant demographic groups, which can lead to an unfair performance disparity across different groups. This is observed especially with underrepresented demographic groups sharing similar voice characteristics. In this work, we investigate the fairness of speaker verification models on controlled datasets with imbalanced gender distributions, providing direct evidence that model performance suffers for underrepresented groups. To mitigate this disparity we propose the group-adapted fusion network (GFN) architecture, a modular architecture based on group embedding adaptation and score fusion. We show that our method alleviates model unfairness by improving speaker verification both overall and for individual groups. Given imbalanced group representation in training, our proposed method achieves overall equal error rate (EER) reduction of 9.6% to 29.0% relative, reduces minority group EER by 13.7% to 18.6%, and results in 20.0% to 25.4% less EER disparity, compared to baselines. The approach is applicable to other types of training data skew in speaker recognition systems.
BibTeX
@inproceedings{shen2022improving,
  author = {Hua Shen and Yuguang Yang and Guoli Sun and Ryan Langman and Eunjung Han and Jasha Droppo and Andreas Stolcke},
  title = {{Improving Fairness in Speaker Verification via Group-Adapted Fusion Network}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=7iXPmaBkIp},
}
Figure