Gender Experts That Leave Before Deployment: Training-Only Scaffolding for Cross-Corpus Speech Models

Speech models for emotion, health, and speaker traits tend to work worse for one sex than the other. The strongest fixes in the literature split the model into separate male and female "experts", which doubles the network that has to run at test time. Those fixes have also mostly been tested inside a single dataset, where a speaker's sex is often tangled up with which corpus and recording setup the audio came from. We propose to keep the experts only during training. A single shared speech encoder is trained alongside small gender-specific adapter branches, each fed only that gender's examples. The adapters' influence is gradually reduced to zero before training ends, so the deployed model is exactly the plain encoder plus one task head. The run-time cost is the same as a naive baseline.

A second training-only constraint keeps the task head geometrically blind to gender and to recording corpus. It does this without erasing that information from the representation. The claim we want to test is narrow. Gender-specific modeling helps because mixed-sex gradients interfere during optimization, not because the model needs gender-specific computation at test time. If that is right, a scaffold that absorbs the interference during training and then dissolves should match run-time experts within a corpus. Because it also separates gender from corpus, it should beat them when tested on corpora never seen in training.

What the literature establishes, and what it leaves open

Two papers motivate the gender side by showing that specialization pays.

  • Bi-encoder for age estimation. Estimation of speaker age and height from speech signal using bi-encoder transformer mixture model places separate male and female 6-layer transformer encoders on top of wav2vec 2.0. On TIMIT it reports age RMSE of 5.54 years (male) and 6.49 years (female). It runs both experts at inference and evaluates only inside TIMIT.
  • GeHirNet for voice pathology. GeHirNet gates sustained-vowel recordings into gender-specific ResNet-50 classifiers. It reports MCC 0.9525 against 0.9041 for a single-stage model. Its centered kernel alignment analysis shows that deeper layers become gender-specific. It pools four corpora (Coswara, ALS, PC-GITA, SVD) that differ in language, disease, and presumably sex balance, and it never holds a corpus out. That is exactly the setting in which "gender expert" and "corpus expert" cannot be told apart.

A third group of papers motivates the fairness side by documenting gender gaps without fixing them.

Gender De-Biasing in Speech Emotion Recognition constrains the design. On MSP-Podcast, adversarially stripping gender collapsed the emotion concordance correlation coefficient (CCC, an agreement score between predicted and annotated values) from 0.684 to 0.518 and trained unstably. A non-adversarial parity penalty kept CCC at 0.661. This is our main reason not to erase gender information.

Two lines of work supply the method.

  • Training-only scaffolding comes from computer vision. DAC-DETR trains a weight-sharing auxiliary decoder to resolve conflicting objectives and then discards it; it reports +3.4 AP on COCO at unchanged inference cost. BatchFormer shows that the auxiliary module only transfers its benefit if the same classifier is supervised both with and without it, the "shared classifier trick". Otherwise the backbone leans on the scaffold and collapses when it is removed. Neither has been applied to speech or to demographic groups.
  • Orthogonal subspaces come from Compositional Feature Alignment. It trains a class head and a domain head under the constraint $W_1 W_2^T = 0$, so domain information sits in directions the class head ignores. It was demonstrated only on vision foundation models and needs domain labels. In speech, corpus identity is a free domain label and speaker sex is usually available as metadata.

The cross-corpus side shows how large the problem is. Normalized modulation spectral features for cross-database voice pathology detection reports detection falling from 94.1% in-corpus to 62.3% when training on MEEI and testing on PdA. Emo-DNA separates emotion from corpus identity across 30 transfer pairs of six emotion corpora but never models speaker sex.

The gap is therefore specific. Nobody has asked whether the benefit of gender experts survives leave-one-corpus-out testing. Nobody has tried to capture that benefit with a scaffold that is gone at deployment.

Core Claim and Mechanism

The deployed model is a pretrained speech encoder $f$ (HuBERT-base or wav2vec 2.0 base) followed by a task head $h$. During training we add, for each gender $s$, a small residual adapter $A_s$: a two-layer bottleneck MLP on the pooled encoder features. A training utterance $x$ from speaker sex $s(x)$ passes through two paths. The shared path computes $h(f(x))$. The scaffold path computes $h(f(x) + \alpha_t A_{s(x)}(f(x)))$, where $\alpha_t$ is a weight that decays from 1 to 0 over training.

Both paths use the same head $h$, following BatchFormer's trick. The adapters can therefore only absorb the part of the gradient that conflicts between sexes. They cannot become a separate classifier the backbone hands work off to.

We also train two linear probes on $f(x)$, never on the adapters: a gender probe with weights $W_g$ and a corpus probe with weights $W_c$. We penalize their overlap with the task head's weights $W_h$. The full objective is

$$ \mathcal{L} = \ell\big(h(f(x)), y\big) + \lambda\, \ell\big(h(f(x) + \alpha_t A_{s(x)}(f(x))), y\big) + \ell_g + \ell_c + \beta \big(\|W_h W_g^T\|_F^2 + \|W_h W_c^T\|_F^2\big), $$

where the terms are:

  • $\ell$ is the task loss (cross-entropy for classification, a CCC loss for continuous attributes);
  • $y$ is the label;
  • $\ell_g$ and $\ell_c$ are the probes' own classification losses;
  • $\lambda$ and $\beta$ are weights;
  • $\|\cdot\|_F$ is the Frobenius norm.

Unlike adversarial debiasing, gender stays linearly decodable from $f(x)$. It is simply placed where $h$ does not read it. At the end of training $\alpha_t = 0$, so the two paths coincide. We delete the adapters and probes and ship $h \circ f$ unchanged.

The mechanism makes three predictions that can each be checked separately:

  1. Inside a corpus, the scaffold should recover most of the gap between a single model and run-time experts.
  2. Under leave-one-corpus-out testing, run-time experts should lose much of their advantage, and the scaffold with the corpus term should beat them.
  3. If the adapters are routed by random groups instead of sex, most of the in-corpus gain should vanish. If it does not vanish, the effect is generic multi-head regularization, not gender.

Research Program

The work has three stages.

Stage 1: measure the confound. We train the standard single model and a run-time gender-expert model. The expert model follows the bi-encoder design, with two expert transformer stacks on a shared encoder and a gender classifier for routing. We compare them in two regimes:

  • the pooled, mixed-corpus split used by GeHirNet;
  • a strict leave-one-corpus-out protocol on the same audio.

The quantity of interest is how much of the expert advantage survives once test corpora are unseen. This stage stands on its own: if the advantage disappears, much of the published motivation for gender experts is corpus leakage.

Stage 2: build and tune the scaffold. The specific choices to settle are:

  • adapter width and the $\alpha_t$ schedule (linear, cosine, step);
  • $\lambda$ and $\beta$;
  • whether to also attach adapters at an intermediate encoder layer, where layer-wise probing suggests the gender split emerges. MVP's layer-5 finding and the depression paper's 9th-block finding tell us intermediate layers matter.

Hyperparameter fragility is a known weakness of auxiliary supervision, as DAC-DETR's top-$k$ thresholds show. So we fix one setting on the source corpora of one task and transfer it unchanged to the others, reporting both tuned and transferred numbers.

Stage 3: analysis. We run linear probes for gender and corpus on the deployed encoder. We measure the cosine overlap between task and gender directions. We run centered kernel alignment between our encoder and each run-time expert, to see whether the single encoder has absorbed what the experts compute. Because treating sex as a binary is a known limitation, we also test a variant that replaces binary routing with soft routing on an estimated vocal-tract-length or $f_0$ score. That variant asks whether the gain tracks anatomy rather than the category.

Baselines

The naive reference is a single shared encoder and head, fine-tuned on pooled source corpora with gender-balanced sampling. It uses the same backbone, epochs, and augmentation as our method. Every method gets the same number of training steps, the same hyperparameter search budget (an identical number of trials on a source-only validation split), and three seeds. Inference parameters, FLOPs, and latency are reported for all of them. Our method must beat this baseline on held-out corpora and on the female-minus-male gap. Matching it would mean the scaffold does nothing.

Run-time gender experts are the strongest existing method in the in-corpus setting. We reimplement the bi-encoder of Gupta et al. on the same encoder. On TIMIT age estimation that paper reports RMSE 5.54 (male) and 6.49 (female), which serves as a sanity anchor for our reimplementation. For the hierarchical variant, GeHirNet reports MCC 0.9525 on its pooled pathology merge. Our method counts as succeeding against experts if it lands within one seed-standard-deviation of them in-corpus at half or less of their inference cost, and beats them under leave-one-corpus-out.

For cross-corpus emotion recognition, the strongest method is Emo-DNA, run from its official code. It reports an average arousal weighted accuracy (WAR) of 74.3% and valence WAR of 58.6% over 30 transfer pairs. Emo-DNA uses unlabeled target-corpus audio for pseudo-labels and distribution matching, whereas our method sees no target data. The fair comparison is therefore twofold:

  • our method against Emo-DNA with its target-adaptation terms disabled;
  • Emo-DNA with our scaffold added, against Emo-DNA alone.

Beating Emo-DNA's full numbers without target audio is not expected and is not the success criterion. Improving it when added, and narrowing its gender gap, is.

The non-adversarial parity regularizer from Gender De-Biasing in Speech Emotion Recognition is also zero-cost at inference and directly targets the gender gap, making it the closest fairness competitor. That paper reports CCC 0.661 and CCC-diff of -0.171 on MSP-Podcast, against a baseline of CCC 0.684 and CCC-diff -0.234. We include its adversarial variant as well, which it reports at CCC 0.518. Our method should hold CCC at or above 0.661 while shrinking the absolute CCC-diff below 0.171.

Ablations isolate each ingredient:

  • Orthogonality alone, with no adapters. This is CFA transplanted to speech and isolates the scaffold's contribution.
  • Adapters alone, with $\beta=0$. This isolates the subspace constraint.
  • Gender-only versus corpus-only orthogonality.
  • Adapters kept at deployment with routing, an upper bound on what the scaffold leaves behind.
  • Random-group routing, the control that tests whether "gender" is doing the work.

Why not just…?

Why not just distill the run-time gender experts into a single student? This is the most serious threat. Distillation is also zero-cost at inference, and in-corpus it may well match the scaffold. The difference shows up cross-corpus. The teacher's experts are specialized on source corpora whose sex balance differs, so the student copies whatever corpus-specific behavior the experts learned. Nothing in distillation separates gender from corpus. We will run distillation with the same training budget, plus a variant that adds our orthogonality term to the student. If plain distillation matches our method under leave-one-corpus-out, the right conclusion is that the adapters are unnecessary and the contribution reduces to the orthogonality term. We will report it that way.

Why not just fine-tune a large foundation model on pooled data with balanced sampling? That is our naive baseline, and existing evidence suggests it leaves the gap in place. Fine-tuned HuBERT still trails on female clinical speakers by 0.05 to 0.1 F1 (depression foundation models). A shared wav2vec 2.0 encoder underperformed gender experts on TIMIT (Gupta et al.). Neither study held out corpora, so Stage 1 is precisely the experiment that could show pooling is enough once leakage is removed. We treat that outcome as a real possible result, not a strawman.

Why not just remove gender adversarially or with a parity penalty? Adversarial removal costs heavily in accuracy and stability on MSP-Podcast (Gender De-Biasing). The parity penalty narrows outcome gaps but builds no gender-aware representation, and it was never tested across corpora. Both are run as baselines. If the parity penalty matches us on held-out corpora, the simpler method wins.

Why not just normalize the signal, for example vocal tract length or subband normalization? Hand-built normalization helps transfer but costs in-corpus information. Normalized modulation spectral features recovers PdA-to-MEEI detection from 51.1% to 76.1% while lowering source mutual information. Feature selection did not stop the cross-database drop in dysarthria classification (Cross-Database Models for the Classification of Dysarthria Presence). We add frequency-warping-style normalization as an input variant of the naive baseline, so its effect is measured rather than assumed.

Proposed Experiments

The benchmarks span four tasks.

Cross-corpus emotion recognition. The six-corpus suite (CASIA, EMODB, EMOVO, IEMOCAP, RAVDESS, SAVEE) uses Emo-DNA's arousal and valence mapping. We run both its 30 pairwise transfers and a leave-one-corpus-out variant with five sources. Metrics are WAR and F1, plus per-sex WAR where speaker metadata allows.

In-corpus fairness. MSP-Podcast v1.3 with speaker-independent splits, reporting CCC, CCC-diff, and TPR-diff against the de-biasing paper's numbers.

Voice pathology across corpora. We run leave-one-corpus-out across SVD, AVFAD, and IPV and report AUC. MVP reports in-corpus AUCs of 95.8%, 96.3%, and 93.6% but no held-out-corpus number; no reported result was found on Lacuna for leave-one-corpus-out on this suite. We also run MEEI↔PdA, anchored by the 62.3% and 76.1% figures above. We rebuild GeHirNet's pooled four-corpus merge both pooled and held-out to directly test the corpus-gender confound.

In-corpus sanity check. TIMIT age estimation, where the claim is parity with the bi-encoder at single-encoder cost.

Three results would decide the proposal.

  1. The confound test. If run-time experts keep their full advantage under leave-one-corpus-out, the "experts overfit corpus" half of the story is wrong. The proposal then reduces to a pure efficiency result.
  2. The random-routing control. If routing adapters by random speaker groups gives the same gain as routing by sex, the mechanism is generic regularization and the gender framing should be dropped.
  3. Distillation with matched orthogonality. This settles whether the dissolving adapters contribute anything beyond a better loss on a single model.

Alongside these we report inference parameters and latency for every method. We also report probe accuracy for gender and corpus on the deployed encoder. Gender should stay decodable while the task head's overlap with the gender direction goes to near zero. That combination is what distinguishes geometric isolation from erasure.