A Study on Bias and Fairness In Deep Speaker Recognition

Amirhossein Hajavi, Ali Etemad

Abstract

As speaker recognition (SR) systems become increasingly integrated into smart devices and security protocols, the risk of algorithmic unfairness—where system performance varies significantly across demographic groups—becomes a critical concern. This study provides a comprehensive audit of fairness in deep SR systems, evaluating how specific neural architectures and loss functions contribute to or mitigate bias. By testing five popular architectures and five loss functions against three distinct statistical fairness definitions, the research identifies that model complexity and the choice of loss function are pivotal in reducing performance disparities across gender and nationality.

Introduction

Speaker recognition technology has transitioned from a niche research area to a ubiquitous feature in modern consumer electronics. However, the rapid deployment of these systems has outpaced the study of their social implications, particularly regarding "algorithmic unfairness." An unfair SR system might authenticate a male user with high precision while frequently failing for a female user, or perform flawlessly for native English speakers while struggling with diverse nationalities. While traditional research has focused on optimizing the Equal Error Rate (EER), this paper investigates how the underlying technical components—the neural network architecture and the mathematical loss function—impact the fairness of the resulting system.

Methodology

Fairness Definitions

The authors evaluate SR systems through the lens of three established statistical fairness definitions:

  1. Statistical Parity: Measures whether the system makes positive decisions (successful authentications) at the same rate across different demographic groups.
  2. Equalized Odds: Requires that the system maintains equal rates of both "correct" positive decisions and "incorrect" positive decisions across groups.
  3. Equal Opportunity: Focuses on the probability of an "incorrect negative" (falsely rejecting a valid user), ensuring this error rate is consistent for everyone.

Experimental Setup

To isolate the effects of technical choices, the study cross-examined several variables:

  • Architectures: Five models were tested, including VGG-M-40, ResNet34L, ResNet34V2, and their "Squeeze-and-Excitation" variants (SEResNet34L and SEResNet34V2).
  • Loss Functions: The study compared classification-based approaches (Standard Softmax, AM-Softmax, and AAM-Softmax) against metric learning-based approaches (Triplet loss and Prototypical loss).

Datasets and Demographics

The research utilized the VoxCeleb ecosystem. Models were trained on VoxCeleb2 (over 7,000 speakers) and evaluated on VoxCeleb1 (1,251 speakers from 11 nationalities). The "protected groups" for fairness testing were defined by gender (female vs. male) and nationality.

Key Insights and Results

The Role of Model Complexity

A significant finding of this study is that model complexity serves as a virtue for fairness. Contrary to the assumption that larger models might overfit to the majority demographic, more sophisticated encoders—specifically ResNet34V2 and SEResNet34V2—aligned more closely with fairness definitions. These larger models appear better equipped to capture the nuanced acoustic characteristics of diverse voices rather than merely memorizing traits of the majority group.

Loss Function Efficacy

The choice of loss function proved to be a powerful lever for controlling bias.

  • AAM-Softmax (Additive Angular Margin Softmax) emerged as the fairest classification-based loss function.
  • Prototypical loss, which compares embeddings to a representative "prototype" of a speaker, significantly outperformed Triplet loss in fairness metrics. The use of multiple prototypes during training allows the model to generalize more effectively across varied vocal samples.

The "English Advantage" and Data Scarcity

Nationality-based analysis revealed a consistent bias in favor of speakers from the "Five Eyes" countries (the US, UK, Canada, Australia, and New Zealand). Despite varying sample sizes in the training set, these speakers experienced lower bias, suggesting that shared linguistic characteristics in English-centric datasets help models generalize more fairly across these specific groups. Conversely, nationalities with fewer representatives in the dataset generally faced higher bias, confirming that data scarcity remains a primary driver of unfairness.

Conclusion and Future Work

This study demonstrates that technical design choices in deep speaker recognition have profound implications for algorithmic fairness. The researchers identified that high-capacity architectures like ResNet34V2, combined with advanced loss functions like AAM-Softmax or Prototypical loss, provide the most equitable performance across gender and nationality.

Future research directions include:

  • Expanding the scope of bias factors to include age and specific regional accents.
  • Investigating data-balancing techniques or the development of "fusion" networks that train separate models for different groups to further mitigate identified biases.
  • Validating these trends across even larger and more diverse datasets beyond the VoxCeleb ecosystem.

Summary updated

Abstract

With the ubiquity of smart devices that use speaker recognition (SR) systems as a means of authenticating individuals and personalizing their services, fairness of SR systems has becomes an important point of focus. In this paper we study the notion of fairness in recent SR systems based on 3 popular and relevant definitions, namely Statistical Parity, Equalized Odds, and Equal Opportunity. We examine 5 popular neural architectures and 5 commonly used loss functions in training SR systems, while evaluating their fairness against gender and nationality groups. Our detailed experiments shed light on this concept and demonstrate that more sophisticated encoder architectures better align with the definitions of fairness. Additionally, we find that the choice of loss functions can significantly impact the bias of SR models.
BibTeX
@inproceedings{hajavi2023study,
  author = {Amirhossein Hajavi and Ali Etemad},
  title = {{A Study on Bias and Fairness In Deep Speaker Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=OUKTt2o8nF},
}
Figure