Finding and Removing the Gender Subnetwork Inside Speech Models' Syllable Prosody

Self-supervised speech models, such as wav2vec 2.0, learn speech representations from raw audio without labels. They now underpin most systems that judge which syllables a speaker stresses or emphasizes. These models also encode speaker gender very strongly. A probe on a multilingual model recovers gender at about 90% accuracy, and a classifier trained only on women's speech generalizes poorly to men's (Gender and Language Identification in Multilingual Models of Speech). We propose to locate the gender signal as a specific set of internal units, using learned binary masks over a frozen model. We then switch those units off and ask two questions. The first is whether syllable-level prominence detection survives and transfers better across speakers. The second is what the removed units actually encode: fixed anatomy, such as pitch level and vocal tract length, or learned, performed gender cues.

The second question is what makes this more than a debiasing exercise. Phoneticians have shown that gendered speech is partly learned behavior. Prepubertal children, whose vocal tracts do not differ by sex, already produce gender-differentiated /s/ (Kinsman et al.). In adults, men's self-ascribed femininity correlates with vowel space size ($r = 0.54$) (Acoustic Correlates of Parental Role and Gender Identity). Neural speech models collapse all of this into one "gender" direction. If gender can be isolated as a removable subnetwork, that subnetwork becomes an instrument for testing what the model treats as gender, including on child speech where anatomy cannot be the explanation.

Core Claim

Gender information used by a frozen speech encoder can be localized to a small, identifiable set of neurons and attention heads. A mask that keeps the units needed for syllable prominence while excluding these gender units should preserve prominence accuracy. It should also cut the train-on-one-gender, test-on-the-other gap more than per-speaker pitch normalization does. Finally, the excluded units should split into two groups. One group tracks anatomical cues (median $F_0$, formant spacing). The other tracks within-syllable dynamics (pitch excursion, duration ratio, spectral emphasis) and predicts perceived gender even in prepubertal children. This last prediction is the scientific payoff, and it is falsifiable.

Relation to Prior Work

The masking method comes from continual learning. There, parameter isolation means reserving separate sets of weights for separate tasks so that learning a new task cannot overwrite an old one. Winning SubNetworks learns task-specific binary masks by attaching a learnable score to each weight and training the scores with a straight-through estimator, a trick that passes gradients through a hard 0/1 threshold. Weights chosen for earlier tasks are then frozen, so earlier tasks are never forgotten (Forget-free Continual Learning with Winning Subnetworks). Functional Task Networks proves that disjoint masks leak no gradient between tasks (Functional Task Networks). Masking whole neurons rather than single weights keeps the structure interpretable and the storage overhead small (Gao et al.). We reuse these tools for a different job. We isolate two concurrent functions inside one frozen network rather than sequential tasks, and we use disjointness as an analysis instrument rather than a guarantee against forgetting. None of these methods has been applied to speech.

Several lines of speech work separate gender or speaker from content, but none does it by isolating weights. Prosody2Vec squeezes prosody through a narrow bottleneck next to a frozen speaker encoder. Speaker information still leaks unless the bottleneck is made very small (Disentangling Prosody Representations). A VQ-VAE anonymizer, which encodes speech into a discrete codebook, drives gender classification to chance (50.01%), but word error rate rises to 73.16% (Protecting Gender and Identity). Gender-ambiguous synthesis controls only utterance-level mean $F_0$ and interpolates between two speakers (Prosody-controllable Gender-ambiguous Speech Synthesis). These papers motivate the proposal by showing the leak and the cost of removing it. They also constrain it: both bottleneck-style removal and representation-level removal can damage downstream utility, so utility must be measured, not assumed.

The prominence task and its syllable granularity come from SupraDoRAL. It pools wav2vec 2.0 features over syllable segments in a joint syllable-word model (JSWPM) and reaches word-prominence F1 of 78.17% (German learners) and 86.56% (Italian learners) on ISLE (SupraDoRAL). An earlier Swedish study showed that syllable-nucleus features beat word-level features (86% vs. 72%) (Prominence detection in Swedish). That study used one male actor and explicitly left cross-gender behavior open, and that open question is the one we address. Classical stress acoustics show that men and women use the same cues to signal stress, but that men have higher spectral emphasis (word stress in English; word stress in Swedish). So some gender signal is entangled with exactly the cues a prominence detector relies on, and removing it is not free.

Research Program

The backbone is a frozen wav2vec 2.0 model, with XLSR-53 as a multilingual replication. Over its feed-forward neurons and attention heads we learn two neuron-level score vectors with straight-through binarization, keeping a fixed fraction $k$ of units per layer.

The gender mask $G$ is trained so that a small head on syllable-pooled masked features predicts speaker sex on LibriTTS. The prominence mask $P$ is trained, together with a JSWPM-style syllable-to-word head, on ISLE word prominence. The isolated model uses $P \setminus G$: every unit selected by $G$ is hard-zeroed during training and testing of the prominence head. Because the backbone is frozen and the units are physically removed, nothing downstream can recover their contribution through retraining. This is the property that separates our approach from erasing gender after the fact at the output.

We sweep $k$ for $G$ to trace a curve: how much gender-probe accuracy falls per unit removed, against how much prominence F1 falls.

The analysis stage regresses the activation of each $G$ unit, averaged over each syllable, onto hand-measured syllable features extracted with the LexStressProcessing-style Praat measurements. These features fall into two families. The anatomical family is median $F_0$ and formant spacing. The dynamic family is semitone $F_0$ variation, nucleus duration relative to neighboring syllables, and spectral emphasis. Each unit is labeled by which family explains more of its variance, and $G$ is split into $G_{anat}$ and $G_{dyn}$.

Both halves are then applied to the LoKiS child corpus (German children aged 6–9, with 7-point listener gender ratings). In these children, natural /z/ spectra did not differ by sex, yet morphing the sibilant shifted perceived gender at $p < .0001$ (How sibilant spectra shape gender perception). We test which half's output predicts listener ratings and responds monotonically to the 20%-step morph continuum. As an explicitly exploratory step, we also test whether $G_{dyn}$ output correlates with GEPAQ femininity among the 15 fathers in AVARE.

Baselines

The naive reference is the unmasked frozen encoder with the same JSWPM-style head. It defines the prominence ceiling and the baseline cross-gender gap. On ISLE, the strongest reported system is SupraDoRAL's JSWPM with wav2vec 2.0, at F1 78.17% (German) and 86.56% (Italian) (SupraDoRAL). We re-implement it on our splits so that every comparison uses identical syllabification (P2TK) and forced alignment (Kaldi). For cross-gender prominence transfer, no reported result was found on Lacuna, so our unmasked baseline sets that number. For the gender side, the reference figures are about 90% probe accuracy on XLSR-53 layer 21 (Gender and Language Identification) and 99.1% for a wav2vec 2.0 layer-3 classifier on LibriTTS (Prosody-controllable Gender-ambiguous Speech Synthesis).

The strongest competitor for the transfer claim is classical normalization. We z-score each speaker's syllable $F_0$, duration, and intensity, convert pitch to semitones, and train on gender-balanced data. It is cheap, interpretable, and attacks the same confound. The second competitor is shared-weight debiasing. We implement both the adversarial gradient-reversal variant and the non-adversarial loss that penalizes gaps in true-positive rate across genders, following Gender De-Biasing in Speech Emotion Recognition. That paper reports overall concordance correlation falling from 0.684 to 0.518 under the adversarial method and to 0.661 under the non-adversarial one. The third competitor is a Prosody2Vec-style bottleneck at matched dimension.

Two ablations isolate our key ingredient, the removal of the specific units in $G$. One removes a random set of units of the same size. The other uses a linear projection that removes the gender direction from the output of the full encoder. All methods share the same frozen backbone, the same heads, and the same data. Each gets the same hyperparameter budget of 30 trials, selected on a held-out validation set.

We count the proposal as winning if three conditions hold together. Prominence F1 must stay within one point of the unmasked model. The syllable-pooled gender probe must fall below 60%. The female-to-male and male-to-female transfer gap must be smaller than the normalization baseline's, with a paired bootstrap confidence interval that excludes zero.

Why not just…?

Why not just normalize pitch per speaker? This is the real threat to the engineering claim, and it may well match us on transfer. The evidence against it fully succeeding is that gender is not carried by pitch alone. Shifting pitch by up to 10% did not change perceived gender ambiguity (Prosody-controllable Gender-ambiguous Speech Synthesis), and spectral emphasis differs by sex even at identical stress levels (word stress in English). Normalizing hand-crafted features also does nothing to the gender signal inside the learned representation. Still, if normalization closes the gap as well as masking does, we will report that. The contribution would then rest on the analysis of $G_{anat}$ versus $G_{dyn}$ and the child-speech tests, which normalization cannot provide.

Why not just train adversarially against gender? The one speech study in the file found adversarial debiasing unstable and costly to accuracy, with concordance correlation falling from 0.684 to 0.518 (Gender De-Biasing in Speech Emotion Recognition). More basically, adversarial training leaves all weights shared, so it yields no inspectable set of units to analyze. Our comparison at matched budget will show whether it also loses on prominence F1.

Why not just erase the gender direction linearly at the output? This is the cleanest alternative, and it is included as an ablation. If it matches $P \setminus G$ on every metric, then localizing gender to units adds nothing for robustness. Even so, the unit split remains the only route to the anatomical-versus-dynamic question.

Why not just use a bottleneck or VQ-VAE? Prior results show either leakage at practical bottleneck sizes (Disentangling Prosody Representations) or a severe loss of utility (Protecting Gender and Identity). We run a matched bottleneck to confirm this on prominence specifically.

Decisive Checks

The engineering claim fails if removing $G$ costs more than one F1 point on ISLE prominence at any setting of $k$ that brings the gender probe below 60%. If so, gender and prominence share units, which is plausible given the stress-acoustics findings. That would be a negative result about how entangled they are inside the model, and we would report it as such.

The scientific claim rests on LoKiS, where anatomy is matched across sexes. If $G_{anat}$ predicts children's perceived-gender ratings as well as $G_{dyn}$ does, the anatomical/dynamic split is not meaningful. If $G_{dyn}$ alone predicts the ratings, the split is supported. Specifically, $G_{dyn}$'s output should rise monotonically along the morph continuum, with the largest steps between 60% and 100% where listeners shifted, while $G_{anat}$'s output stays flat.

A further control rules out a trivial explanation: units selected by a random mask should show neither effect. The AVARE femininity correlation is reported only as exploratory, given 15 male speakers.