Do Speech Quality Predictors Hear Male and Female Consonants Equally?

Listening tests, where people rate audio on a 1-to-5 scale and the ratings are averaged into a Mean Opinion Score (MOS), are slow and expensive. So researchers in speech synthesis and speech enhancement increasingly let a neural network that predicts MOS stand in for the listeners. These MOS predictors are judged only by how well their scores correlate with human scores over a whole corpus. Nobody has checked whether they are equally accurate for male and female voices. There is a concrete reason to suspect they are not. Speech enhancement is known to damage different speech sounds differently for each gender: male plosives (short bursts such as /p/, /t/, /k/) carry low-frequency energy that common noise masks, and they come out with more processing artifacts than female plosives. Meanwhile, MOS predictors average frame-level scores uniformly over time, so a brief consonant dropout is swamped by the surrounding vowels. If a predictor is deaf to exactly the damage that differs by gender, then any fairness study that uses it in place of listeners will measure the wrong gap.

This note proposes two linked contributions. The first is a controlled audit that measures, against a small but targeted human listening test, how strongly MOS predictors react to damage confined to one class of speech sound, separately for male and female speakers. The second is a predictor that conditions its internal feature normalization and its time pooling on soft estimates of which broad class of speech sound is present. The central claim is narrow and testable. Speaker-gender errors in current MOS predictors are not a constant offset that per-gender recalibration could remove. They depend on which class of speech sound is degraded. Making the predictor's normalization and pooling aware of sound class shrinks this gender-by-sound-class mismatch, while still matching the strongest existing predictors on standard corpus-level correlation.

What exists and what is missing

The research file contains three relevant strands, and none of them asks this question. The motivating evidence comes from phoneme-scale studies of enhancement. A Phoneme-Scale Assessment of Multichannel Speech Enhancement Algorithms splits enhancement performance by phoneme using forced alignment and finds that plosives are the weak point of every system tested. Its follow-up, Evaluating Multichannel Speech Enhancement Algorithms at the Phoneme Scale Across Genders, repeats the analysis by gender. It reports a 2.89 dB output signal-to-interference gap on vowels in favor of female speech, significantly more artifacts on male plosives ($p=0.02$), and an 8.83-point absolute word error rate penalty for male speech at -5 dB. Both papers are purely diagnostic and use intrusive signal metrics, meaning metrics that need a clean reference recording. Neither asks what a listener, or a predictor of listeners, would conclude.

The method comes from phoneme-dependent normalization. Phoneme-based Distribution Regularization for Speech Enhancement (PbDrNet) feeds soft phoneme probabilities into a layer that rescales and shifts intermediate features, in the style of batch normalization, and improves PESQ from 2.72 to 2.85. Earlier work does the same thing with classical tools: Buera et al. use phoneme-specific compensation vectors, and Chng et al. use histogram equalization per manner of articulation. All of these apply the idea to recognition or enhancement. None applies it to a quality assessor.

The constraint comes from MOS prediction itself. UTMOS, the VoiceMOS 2022 winner, appended phoneme embeddings from a speech recognizer to self-supervised features and found the gain negligible on the main track. Its authors attribute this to the backbone already encoding phonetic content. We take that result seriously. It rules out "add phoneme tokens" as a contribution. It does not rule out using phoneme class to change how features are normalized and how frames are weighted, which is a different operation. Listener-side modeling, as in MBNet and DDOS, models the rater's identity and never the speaker's demographics. The VoiceMOS Challenge 2022 collected listener demographics that no team used and reported no speaker gender composition. The VoiceMOS Challenge 2024 found that predictors track background noise well but track intrinsic speech distortion poorly. It did not ask where in the signal that distortion lives.

One more paper constrains the design. Twists, Humps, and Pebbles finds that in speech recognizers the gender error gap shrinks when gender is more extractable from hidden states ($\rho=-0.65$). It also finds that utterance-level pitch and intensity do not explain the gaps. We read this as two warnings. First, the predictor should not be made gender-blind, so we do not use adversarial removal of gender. Second, acoustic explanations need to be local to sound classes rather than averaged over sentences. That second point is exactly where the Monir et al. findings and Savoldi et al.'s null result can both be true.

Core Claim

We define a predictor's sensitivity to targeted damage as follows. A degradation is confined to broad sound class $c$ (vowels, nasals, fricatives, plosives) in speech from speaker group $g$. Let $\Delta_H(c,g)$ be the drop in human MOS from the clean to the damaged version, and let $\Delta_P(c,g)$ be the drop in the predictor's score. The quantity of interest is the sensitivity mismatch

$$M(c,g) = \Delta_P(c,g) - \Delta_H(c,g),$$

together with its gender disparity $D(c) = M(c,\text{female}) - M(c,\text{male})$.

A predictor can have excellent corpus-level correlation and still have large $|D(c)|$ for plosives. Two facts make this likely. Plosive frames are few, and the gender difference in plosive damage is the largest one documented. The claim has two parts. Existing predictors show nonzero $D(c)$ that varies in size or sign across classes. A sound-class-aware predictor reduces $\sum_c |D(c)|$ without losing standard correlation. If $D(c)$ turns out to be roughly the same across classes, then a per-gender affine correction is enough, and the method half of this proposal is unnecessary. The audit is designed to reveal that outcome rather than hide it.

Research Program

Speaker gender labels. We first attach speaker gender to existing MOS corpora. In VCC 2018 the source and target speakers are a small, fixed set, and their genders can be recovered from the challenge design and checked by listening. For BVCC we label each system's output with an off-the-shelf gender classifier and hand-verify a stratified sample. Disagreements are reported, and ambiguous items are dropped rather than forced into a category. On these labels we compute ordinary per-gender error statistics for existing predictors: MSE, SRCC, and the mean signed residual. This is the cheap part of the audit, and it alone fills the gap that no MOS predictor has been disaggregated by speaker gender.

Controlled stimuli and a targeted listening test. The decisive part needs stimuli where we know where the damage is. We take gender-balanced LibriSpeech test speakers, aligned with the Montreal Forced Aligner as in the Monir et al. studies. We then apply degradations confined to one broad class: burst smearing and low-frequency masking on plosives, high-band attenuation on fricatives, and spectral smoothing on vowels. Each is applied at three severities, calibrated so that the local distortion energy matches across genders. As a second, realistic stimulus set we add outputs of single-channel enhancement systems on noisy mixtures, including a PbDrNet-style model.

A crowdsourced test then collects ratings on this set, with roughly eight ratings per stimulus and listener demographics recorded as in BVCC. The point is that the listening bottleneck is spent only where it is informative. The test does not reproduce a corpus. It is a measurement instrument for $\Delta_H(c,g)$.

The proposed predictor. We start from an SSL-MOS-style predictor, that is, a fine-tuned self-supervised speech encoder with a regression head. A small frozen phoneme recognizer outputs soft posteriors $\mathbf{p}_t$ over the broad sound classes at frame $t$. Because the text is known for TTS and VC, forced alignment is available as a cross-check, but soft posteriors avoid hard alignment failures on degraded speech.

Two changes are made. The first is class-conditioned modulation in the style of PbDrNet. At a middle encoder layer, the normalized feature $\bar{\mathbf{h}}_t$ is replaced by $\gamma(\mathbf{p}_t)\odot\bar{\mathbf{h}}_t+\beta(\mathbf{p}_t)$, where $\gamma$ and $\beta$ are small learned maps from class posteriors to per-channel scale and shift. This lets the predictor judge a plosive frame against what plosives typically look like, instead of against vowel-dominated statistics. Gender information is untouched, consistent with Savoldi et al. The second change is class-balanced pooling. The utterance score averages frame scores first within each class, weighted by posterior mass, and then across classes with learned weights. Thirty vowel frames can then no longer outvote one destroyed burst. Training uses the standard corpora and no gender labels, so the method needs no demographic metadata at test time.

Interpretation. For every predictor we compute $M(c,g)$ and $D(c)$ with bootstrap confidence intervals over speakers and listeners. We also check whether the class-by-gender residual structure found on controlled stimuli predicts the residual structure on real VC and enhancement outputs. That check is what connects the lab measurement to the corpora people actually use.

Baselines

The naive reference point is the gap that exists with no intervention. We measure $D(c)$ and the per-gender residuals for an SSL-MOS predictor trained on BVCC exactly as in VoiceMOS 2022, and for the older MOSNet, whose uniform frame averaging defines the problem. Any proposed change must move $D(c)$ relative to these.

The strongest existing method is UTMOS. It is the best documented predictor on the shared benchmark, and it already includes phoneme information, so it is the most direct competitor to the claim that sound-class awareness matters. On BVCC main track it reports utterance-level MSE 0.165 and SRCC 0.897 (UTMOS). For VCC 2018 the reference is MBNet's system-level SRCC of 0.949 (MBNet). We will run UTMOS's strong-learner configuration and our model on the same backbone checkpoint, with the same training data and the same number of hyperparameter trials, selected on the same validation split. Beating it means two things together: an utterance-level SRCC on BVCC within the run-to-run noise of UTMOS's reproduced value, and a statistically lower $\sum_c|D(c)|$ on the controlled set. For the gender mismatch metric itself, no reported result was found on Lacuna, for any predictor.

The fairness-specific baselines are per-gender post-hoc affine calibration of predictions, fit on held-out labeled data, and gender-balanced resampling during fine-tuning. Both get the same validation budget as our method.

The key ablations isolate the ingredient. One variant has modulation but uniform pooling, and another has class-balanced pooling but no modulation. A third replaces the class posteriors with UTMOS-style concatenated phoneme embeddings at matched parameter count; this tests whether conditioning the normalization, rather than adding phoneme features, is what matters. A fourth uses shuffled class posteriors, to rule out the extra capacity itself as the cause.

Why not just…?

Why not just recalibrate predictions separately for each gender? This is the real threat. If the audit shows $D(c)$ roughly equal across classes, a per-gender offset and scale solves the problem with no new model, and we will say so. Calibration fails only if the mismatch differs in size or sign between classes within a gender. For example, the predictor might under-react to male plosive damage while over-reacting to male vowel damage. The Monir et al. finding that the gender gap flips direction between vowel signal-to-interference ratio and plosive artifacts makes that pattern plausible but not certain. The controlled test measures it directly. Calibration also needs gender labels at test time, which our method does not.

Why not just rebalance the training data by gender? Rebalancing changes how often each group is seen. It does not change the fact that uniform pooling dilutes brief consonant frames for everyone. If the mismatch comes from pooling dilution interacting with where each gender's damage occurs, rebalancing will leave $D(\text{plosive})$ in place. The resampling baseline tests exactly this. Rebalancing is also hard to do well here, because BVCC reports no speaker gender composition.

Why not just rely on a bigger self-supervised backbone, as UTMOS's null phoneme result suggests? UTMOS shows that phoneme tokens add little to corpus-level correlation. It never measured class-specific sensitivity or gender-disaggregated error, the quantities at issue here. FaiST found significant demographic bias in all 18 recognizers it tested, including large self-supervised and Whisper models, and VoiceMOS 2024 found weak tracking of speech distortion even with strong backbones. Neither result proves that scale fails for MOS fairness, so we include the largest backbone that fits the budget as an additional baseline, with and without our changes.

Why not just use attention pooling? Learned attention can up-weight anomalous frames, as in Jing Wang et al.. It may recover much of the pooling benefit. That is why the pooling-only ablation is matched against an attention-pooling variant. If attention matches class-balanced pooling on $D(c)$, the contribution narrows to the audit and the modulation.

Decisive Checks

Three outcomes settle the question. The first is whether existing predictors show a gender mismatch $D(c)$ whose confidence interval excludes zero for at least one class, and whether it varies across classes. If not, the method is moot and the audit is the publishable result. The second is whether our predictor lowers $\sum_c|D(c)|$ more than per-gender calibration, rebalancing, and attention pooling do, while staying within noise of the reproduced UTMOS correlation on BVCC. The third is whether the residual structure measured on controlled stimuli predicts gender-disaggregated residuals on real VC and enhancement outputs. As a final practical test, we re-score the gender gap of enhancement systems once with human ratings and once with each predictor, and report whether a predictor would have reversed or hidden the gap that the Monir et al. analyses found.