TL;DR
This paper argues that fairness in Automatic Speech Recognition (ASR) is a generalization problem, not a one-size-fits-all solution. They introduce a novel contrastive learning approach that promotes gender-invariant representations, showing it offers a practical, generalizable way to improve fairness across different domains without altering training data.
What Problem Does This Paper Address?
Modern ASR systems, like OpenAI's Whisper, are increasingly deployed in critical applications such as healthcare and education. While these powerful pre-trained models offer impressive overall accuracy, they often exhibit significant performance disparities across different demographic groups, particularly concerning gender. Existing fairness interventions, often developed for specific tasks or datasets, may not generalize well when the model encounters new data distributions (domain shift). This means a model that appears fair in its training environment might become unfair in real-world scenarios, undermining trust and equitable access to technology. The paper aims to address this by investigating fairness as a generalization problem and proposing a method that maintains fairness across diverse data.
Methods
The researchers explore four fairness-aware fine-tuning strategies for the Whisper-small model:
- Basic Fine-Tuning (FT): Standard fine-tuning on the original dataset.
- Demographic Rebalancing (FT-Balanced): Fine-tuning on a subset of the data that is balanced for gender.
- Voice Conversion Augmentation (FT-Augmented): Augmenting the training data by using a voice conversion model (XTTS-v2) to synthesize speech with the same text content but from a different gender's voice. This aims to expose the model to identical transcriptions spoken by both male and female voices.
- Contrastive Learning (CL): A novel approach where they introduce a contrastive loss term alongside the standard cross-entropy loss. This contrastive loss encourages the model to produce similar embeddings for utterances with the same linguistic content, regardless of the speaker's gender. Positive pairs are created using gender-swapped utterances (from FT-Augmented), while negative pairs are utterances with different content. This method aims to learn gender-invariant representations without modifying the underlying training data distribution.
Datasets
- Fair-Speech: Used for both training and in-domain evaluation. This dataset is designed for fairness research in speech processing and includes speaker demographic attributes like gender.
- LibriSpeech (test-other subset): A standard benchmark for evaluating ASR robustness under challenging acoustic and linguistic conditions, derived from audiobooks.
- EdAcc (Edinburgh International Accents of English Corpus): A dataset designed for research on accent variation and robustness, featuring recordings from virtual calls with diverse speakers.
- CognoSpeak: An ongoing project collecting audio and video recordings of individuals with cognitive decline, used here for evaluating ASR performance in a task-oriented clinical interview setting.
Key Insights and Ideas
- Fairness is a Generalization Problem: The most striking insight is that fairness gains achieved on an in-domain dataset do not necessarily translate to out-of-domain performance. A model that looks fair locally might be unfair globally.
- In-Domain Fairness is Misleading: Basic fine-tuning (FT) can lead to the lowest gender gap in-domain but can perform the worst out-of-domain, highlighting the risk of overfitting to the training distribution.
- Contrastive Learning Offers a Promising Path: The proposed contrastive learning method (CL) demonstrates generalizable fairness improvements without needing to alter the training data distribution, making it a practical and scalable solution.
- Metric Choice Matters: The choice of evaluation metric significantly impacts conclusions. While the absolute gender gap is common, metrics like Word Error Rate Reduction (WERR) can reveal that apparent fairness gains might mask degraded overall performance.
- Initial Disparities Influence Effectiveness: Fairness interventions tend to be more effective on datasets with larger initial gender disparities.
Key Results
- The FT-Balanced and FT-Augmented models showed more robust and fair performance across domains compared to basic FT.
- The CL(a=0.2) model achieved the lowest WERs on the EdAcc dataset and outperformed basic FT on all out-of-domain sets, demonstrating improved generalization.
- While FT significantly reduced the gender gap on Fair-Speech (from 8.17% to 3.17%), this improvement was not sustained on out-of-domain datasets.
- The CL(a=0.05) model, despite achieving a small gender gap on EdAcc, worsened performance for the disadvantaged group by over 20%, underscoring the importance of considering performance alongside disparity.
Limitations and Future Work
- Binary Gender Labels: The study's evaluation datasets primarily use binary gender labels, which do not capture the full spectrum of gender identities and may limit the generalizability of the fairness analysis.
- Limited Metrics: The research focused on a set of widely adopted fairness metrics. Exploring other metrics could provide deeper insights.
- Gender-Centric Focus: The current analysis is limited to gender-based fairness. Future work should extend to other demographic factors like age, race, and accent.
- Scalability of Contrastive Learning: The authors suggest that contrastive learning is a promising direction for scalable, generalizable fairness, particularly for multi-attribute fairness where demographic labels are available. They propose extending this to other protected attributes by treating same-content utterances across these attributes as positive pairs.
Summary updated