On the Encoding of Gender in Transformer-based ASR Representations

Aravind Krishnan, Badr M. Abdullah, Dietrich Klakow

Abstract

Modern Automatic Speech Recognition (ASR) models often exhibit performance disparities across different speaker demographics, particularly gender. This research investigates the internal representation of gender within Transformer-based models like Wav2Vec2 and HuBERT. By employing linear erasure and amnesic probing, the study demonstrates that while gender information is deeply embedded in these models, it is not functionally necessary for the transcription task. The findings suggest that linguistic and gender-related information occupy largely orthogonal subspaces, allowing for the creation of gender-neutral representations that maintain high accuracy while enhancing speaker privacy.

Introduction

As Automatic Speech Recognition (ASR) systems become ubiquitous, addressing algorithmic bias has become a critical priority in AI safety and fairness. It is well-documented that ASR systems often perform significantly better for male speakers than female speakers. However, most existing literature focuses on measuring these performance gaps—the "symptoms" of bias—rather than the underlying "pathology" of how gender is encoded within the neural network's architecture.

This paper addresses a fundamental question: If a model retains a speaker's gender information, does it utilize that information to improve transcription accuracy, or is it extraneous data that could potentially lead to biased outcomes? By attempting to "scrub" gender information from the model's internal layers, the researchers test whether the model's performance relies on these sociolinguistic cues.

Methodology

Linear Erasure and Concept Scrubbing

The researchers utilized Least Squares Concept Erasure (LEACE), a method designed to transform internal embeddings such that a linear classifier can no longer distinguish between gender categories. This transformation is optimized to change the original embeddings as little as possible while ensuring the target concept (gender) is effectively neutralized.

To prevent the model from "re-learning" gender cues in subsequent layers, the authors implemented Concept Scrubbing. This involves a layer-by-layer intervention where the erasure is applied at the output of every single transformer layer in a cascade. This rigorous approach ensures that the gender-neutral state is maintained throughout the network's depth.

Probing Techniques

Two distinct probing methods were used to analyze the representations:

  1. Snapshot Probing: Rather than analyzing the average representation of an entire audio clip, the researchers took ten equally spaced "snapshots" across the time dimension. This allowed for the observation of where gender information is most concentrated within the audio signal.
  2. Amnesic Probing: This behavioral test involves removing a specific property (gender) and observing the impact on the primary task (transcription). If performance remains stable, it can be concluded that the model is not strictly dependent on that property.

Experimental Setup

The study utilized two standard benchmarks:

  • Librispeech (100h): A large-scale dataset of read English audiobooks, serving as the primary in-domain dataset for training and evaluation.
  • TIMIT: A phonetically rich corpus of 630 speakers, used as an out-of-domain test to evaluate the generalization of the scrubbing techniques.

Note: The researchers acknowledge the limitation of using binary (Male/Female) gender labels, as provided by these legacy datasets.

Key Insights and Results

The "Autoencoder" Tendency

A significant finding of the study is the spatial distribution of gender information in the final layers of the models. Gender cues are not spread evenly; instead, they migrate to the first and last frames of the audio clip. This mimics the behavior of an autoencoder, where the model "tucks away" speaker identity at the temporal edges to clear the middle frames for intensive linguistic processing. Snapshot probing revealed that in the last four layers, gender awareness in the intermediate snapshots falls to near-random levels.

Subspace Orthogonality

The results suggest that the "linguistic space" and "gender space" within the final layers are largely orthogonal. Because these spaces are separate, it is possible to delete gender information without damaging the linguistic content required for transcription.

Performance Impact

The intervention had a negligible impact on the Word Error Rate (WER). For Wav2Vec2 on the Librispeech dataset, the WER only shifted from 2.76% to 2.77% after gender scrubbing. This proves that gender-neutral embeddings are entirely viable for high-performance ASR. Furthermore, in the final layers of both Wav2Vec2 and HuBERT, linear probes dropped to near-random accuracy (approximately 50%), indicating successful anonymization.

Early Layer Resilience

While the final layers were easily scrubbed, the early layers of the Transformer showed a high capacity for "recovering" gender information even after linear erasure. This is likely due to the high parametric capacity of early layers and their use of non-linear operations to reconstruct missing signals from the raw input.

Conclusion and Future Work

This research demonstrates that gender information is a separable component of ASR representations. By successfully scrubbing this information with minimal performance loss, the authors provide a pathway toward more private and potentially less biased speech technologies.

Future work aims to move beyond binary gender constraints and explore "speaker-blind" embeddings. Such representations would aim to remove all individual speaker characteristics, further enhancing privacy and mitigating various forms of speaker-based bias in automated systems.

Summary updated

BibTeX
@inproceedings{krishnan2024encoding,
  author = {Aravind Krishnan and Badr M. Abdullah and Dietrich Klakow},
  title = {{On the Encoding of Gender in Transformer-based ASR Representations}},
  booktitle = {Interspeech},
  year = {2024},
  doi = {10.21437/interspeech.2024-2209},
}
Figure