Speech recognizers trained on male-dominated data make more errors on some speaker groups than others. Which group loses varies by language and model in ways nobody can predict. This proposal borrows a tool from structural biology to explain why. When an enzyme changes shape, crystallographers superimpose its two states and ask which parts stay rigid and which parts bend. They find a stable core held together by conserved internal hydrogen bonds, which work like internal staples, plus a few hinges where the motion happens. We apply the same superposition to the internal geometry of a speech model. Within one language we compare how the model arranges its sound categories for female, male and non-binary speakers. We identify a conserved core of distance relationships, the "staples," and the regions that bend, the "hinges." We then test two claims. The first is that the size of the bent region predicts a language's gender error gap better than current explanations do. The second is that a fine-tuning loss holding only the staples fixed shrinks the gap without erasing gender information from the model.
The alternatives already in the literature are either suppressing gender or rebalancing data, and each has a limitation this approach avoids. Twists, Humps, and Pebbles found that across 19 languages, languages where Whisper and SeamlessM4T encode speaker gender more clearly have smaller female error gaps (ρ = −0.65). The same study found that pitch and loudness do not explain the gaps. Together these results rule out the standard fix of making representations blind to gender, and they point away from the raw acoustic signal toward the model's internal organization. Resampling methods such as SMOTE (The Balancing Act) change the data but say nothing about which internal structure is broken. Our method keeps gender visible and constrains only the relations between content categories that should be shared across speakers.
Core Claim
A speech encoder's representation of one language can be viewed as a "molecule." Its "atoms" are the average hidden-state vectors, or centroids, for each phone or character class at a given layer, computed separately for each speaker group. Its "shape" is the matrix of pairwise distances between those centroids. That matrix does not change if the whole space shifts along a gender direction, so gender can remain linearly decodable while the distances stay fixed. Our claim is that gender gaps arise when the minority group's shape bends away from the majority's in particular regions, the hinges. The claim is not that the groups sit far apart overall. The rest of the structure, the conserved core, stays fixed. We further claim that holding the core fixed during fine-tuning, and leaving the hinges free, closes the gap better than either pinning every relation or pinning none.
The analysis method is taken directly from Analysis of protein loop closure. That work used "fit-all," "sieve-fit" and "fit-refit" superpositions of the open and closed states of lactate dehydrogenase. The procedure repeatedly fits the two states, discards the worst-matching atoms, and refits, until a static core remains. It showed a roughly 10 Å loop motion produced by two hinges, while five transverse hydrogen bonds inside the loop were conserved across both states and across 17 sequences. That paper supplies our procedure and our central hypothesis: the useful signal is the split between what stays conserved and what moves.
A second chemistry paper informs how we decide which pairs count as staples. Recognizing molecular patterns by machine learning learned a definition of a hydrogen bond from geometry instead of using a fixed distance-and-angle cutoff. Following it, we choose the core with a data-driven stability criterion across bootstrap resamples of speakers, not a hand-set threshold.
Research Program
The first part of the work is diagnostic. For each language and layer, we compute class centroids per gender group using time alignments from a forced aligner, a model that marks where each character or phone starts and ends in the audio. We then build the distance matrix for each group and run sieve-fit between the majority group and each minority group. This yields a core set $C$ of centroid pairs whose distances agree within bootstrap noise, and a complementary hinge set. The quantity we study is the hinge fraction, meaning the share of pairs outside $C$, together with where the hinges fall in the layer stack.
We will test whether the hinge fraction predicts the per-language female gap and the "Other" gap on Common Voice across the 19 languages studied by Attanasio et al. The key test is a partial correlation that controls for their gender-extractability score. If the hinge fraction adds no predictive power beyond extractability, the diagnostic contributes nothing new and we will report that result.
The second part is an intervention. During fine-tuning we add a staple loss:
$$ \mathcal{L}_{\text{staple}} = \sum_{(i,j)\in C} \left( d^{\,m}_{ij} - \operatorname{sg}\!\left[d^{\,r}_{ij}\right] \right)^2 $$
Here $d^{\,m}_{ij}$ is the distance between the minority group's centroids for classes $i$ and $j$, computed on the current batch. $d^{\,r}_{ij}$ is the same distance for the reference group, pooled across all groups. $\operatorname{sg}$ is a stop-gradient, so the reference group's distances are not pushed toward the minority's. $C$ is the sieve-fit core, re-estimated every few thousand steps. The loss is added to the usual transcription loss with a single weight. Because the loss constrains only distances, it cannot remove gender information. We verify this by probing gender after fine-tuning, and we expect extractability to stay the same or rise.
One chemistry result shows that the choice of which pairs to staple matters, not only whether to staple. In the calix[4]arene crystal structures, hydrogen bonds on the lower rim (Luo et al.) produce a symmetric cone. Hydrogen bonds on the upper rim (Hu et al.) instead pull the macrocycle into a pinched, uneven shape. Our ablations are built around the corresponding question: does fixing the core help, while fixing every pair, which includes the hinges, harms the model?
Baselines
The naive reference point is the pretrained model with no adaptation, evaluated zero-shot. On the podcast gender-by-dialect set, Modeling Gender and Dialect Bias in Automatic Speech Recognition reports Whisper-medium as the strongest zero-shot system at 24.8% WER. On colloquial Arabic, Casablanca reports Whisper-large-v3 at 63.00% average WER.
The strongest existing method on our main benchmark is the same paper's fine-tuning of a Wav2vec2 Conformer on a gender-and-dialect-balanced subset. This lowered WER by 17.3% and reached 17.8% WER on the held-out test set. It is the strongest competitor because it directly addresses gender imbalance with balanced data, which is exactly what our loss must improve on. We will reproduce it on the same balanced subset and use it as the starting point for our method. Both arms receive the same number of training steps and the same hyperparameter search budget, with our single loss weight included in that shared budget.
The second baseline is SMOTE applied to encoder features, following The Balancing Act. That paper reports that SMOTE balanced male and female WER across all tested systems on the CCD V2 Portuguese data. It gives no per-model WER figures, so we have no number to match. We will rerun it with the same budget.
The third baseline is adversarial gender removal, a gradient-reversal classifier on the encoder. It is the natural foil given the evidence that gender-blind representations make gaps worse.
Two ablations isolate what is new in our method. The first is a global staple that fixes all pairwise distances. The second is a random staple that fixes a random subset of pairs the same size as $C$.
Beating the baselines means lowering the maximum-minus-minimum group WER below the balanced fine-tuning result without raising overall WER by more than 0.5 absolute points. On the podcast set, the overall WER must also stay at or below the reproduced 17.8%. Separately, the core-only loss must outperform both the global and random staples.
Why not just…?
Why not just fine-tune on balanced data? This is the most serious threat, since Harris et al.'s 17.3% reduction is large. Balanced fine-tuning, however, depends on having minority data. For "Other"-tagged Common Voice speakers the data is sparse, and the penalty reaches 52% in Catalan. Our loss uses the majority's structure as a scaffold, so it should help most where minority data is scarcest. We will test this directly by subsampling minority training data at 100%, 25% and 5%. If our method shows no advantage over balanced fine-tuning at any of these levels, the intervention part of the proposal fails, and only the diagnostic remains.
Why not just use SMOTE? SMOTE creates new points by interpolating linearly between existing minority examples. It helped on scripted Portuguese. Kulkarni et al. themselves list interpolation of high-dimensional, time-varying features from spontaneous speech as an open problem, and our main test set is spontaneous podcast speech. Running SMOTE with the same budget on that set settles the question empirically.
Why not just compare representations with a global similarity score like CKA or Procrustes alignment? If a single global score predicts the gaps as well as the hinge fraction does, and if the global-staple ablation matches the core-only loss, then separating core from hinges adds nothing. Both ablations are designed to reveal exactly that outcome.
Why not just condition the model on gender explicitly? The gender-aware branching approach of GeHirNet requires a gender label at inference time and assumes binary categories. That rules it out for non-binary speakers and for deployment without labels. Our loss needs group labels only during training.
Proposed Experiments
We run the diagnostic on Whisper-large-v3 and SeamlessM4T v2 over Common Voice, Fleurs and VoxPopuli. The decisive check is the partial correlation between hinge fraction and gender gap after controlling for gender extractability.
We run the intervention on three datasets. The first is the Harris podcast set, using gender-by-dialect cells with CORAAL as out-of-domain evaluation. This tests whether stapling also reduces the "intersectional flip," where the gender advantage reverses in minority dialects, and the collapse of dialect forms into standard English. The second is CCD V2 Portuguese, compared against SMOTE. The third is Casablanca, where more than 60% of speech is male, compared against its reported Algerian fine-tuning improvement from 83.49% to 60.33% WER.
We report per-group WER, the worst-group gap, gender-probe F1 after training, and hallucination-loop counts. For FaiST we report the direction of mixed-effect regression coefficients. We found no reported numerical coefficients for FaiST on Lacuna, so these results are exploratory.
The proposal succeeds if two things hold together: the core-only loss narrows the gap more than the global staple, the random staple and balanced fine-tuning in the low-data settings, and gender remains decodable from the model afterwards.
References
- Twists, Humps, and Pebbles: Multilingual Speech Recognition Models Exhibit Gender Performance Gaps
- The Balancing Act: Unmasking and Alleviating ASR Biases in Portuguese
- Analysis of protein loop closure
- Recognizing molecular patterns by machine learning: An agnostic structural definition of the hydrogen bond
- Crystal structure of 5,11,17,23-tetrabromo-25,27-dihydroxy- 26,28-dimethoxycalix[4]arene, C30H24Br4O4
- Crystal structure of 5,17-bis(N-tert-butylhydroxyamine)-25,26,27,28- tetrapropoxycalix[4]arene, C48H66N2O6
- Modeling Gender and Dialect Bias in Automatic Speech Recognition
- Casablanca: Data and Models for Multidialectal Arabic Speech Recognition
- GeHirNet: A Gender-Aware Hierarchical Model for Voice Pathology Classification
- FaiST: A Benchmark Dataset for Fairness in Speech Technology