How Much of Speaker Normalization Is Erasing the Diagnosis?

Voice-based screening for conditions like depression begins by measuring pitch, loudness and spectral shape from speech. These raw numbers depend heavily on who is speaking. A typical male voice sits lower in pitch than a typical female one, and phones differ in their microphones. So nearly every clinical speech pipeline "normalizes" the features: it re-centres and rescales each measurement relative to the speaker, their gender group, or their device. The step demonstrably helps. On a large smartphone depression corpus, gender-specific normalization raised depressed-class F1 from 0.290 to 0.396 (Depression Detection from Short Utterances via Diverse Smartphones). What nobody has measured is why it helps.

Three explanations are possible, and they have very different clinical consequences. Normalization could be removing nuisance anatomy. It could be quietly exploiting differences in how often each gender is depressed. Or it could be improving some features while erasing the biomarker in others, such as a flattened pitch range. This proposal runs a controlled component-contribution study of the normalization step itself. It borrows the counterfactual controls from ablation research: blindfold baselines, matched random controls, and exact attribution over feature families. The goal is to split the gain into these three parts. The same machinery then tests one prediction. Normalizing against each speaker's own statistics should help when the label changes within a speaker, as with eating condition or emotion, and should hurt when the label is a fixed property of the speaker, as with depression.

Core Claim

Normalization is a set of separate operations whose contributions can be measured, not one front-end switch. We claim three things. First, part of the reported gain from group normalization comes from how the reference statistics absorb the condition itself. Second, computing those statistics only from healthy speakers in the training fold removes this contamination without losing the anatomical correction. Third, the benefit of normalizing is sign-inconsistent across feature families, so exact Shapley attribution over a handful of families yields a selective normalization policy. Shapley attribution here means averaging each family's marginal effect over every combination of the other families. We also expect this policy to transfer across regimes in a predictable way.

The contamination mechanism is simple enough to state exactly. Let $\mu_g$ be the mean of a feature in gender group $g$, let $\mu_g^{h}$ be its mean among healthy members of that group, let $p_g$ be the fraction of depressed speakers in the group, and let $\delta$ be the average shift depression induces in the feature. Then

$$\mu_g = \mu_g^{h} + p_g\,\delta .$$

Subtracting $\mu_g$ therefore removes a slice of the very signal being detected, and the slice is larger in whichever group has higher prevalence. Subtracting $\mu_g^{h}$ removes only the anatomical offset. Per-speaker normalization is the extreme case, with $p=1$ for a depressed speaker. It subtracts the speaker's whole trait-level shift. Z-scoring pitch frames by the speaker's own standard deviation also sets pitch variability to roughly one for everyone, and pitch variability is precisely the monotony cue clinicians care about.

Relation to Prior Work

The motivation comes from the normalization literature itself. The smartphone depression study compared mean, variance, histogram and condition-specific normalizations. It found gender beating manufacturer normalization (0.396 vs 0.360), but it normalized the whole feature matrix at once. It could not say which features gained, which lost, or whether prevalence differences explain the gain. Towards interpretable speech biomarkers: exploring MFCCs showed physically that the second cepstral coefficient tracks glottal aspiration noise and runs higher in men because of their lower pitch. That makes it a concrete family where sex normalization should help, and where per-speaker normalization should destroy the signal.

Two papers show that relative, within-speaker measurements win when the label changes within the speaker. The work on stress responses in children with autism got its best diagnostic results from baseline-to-task changes. Detecting Vocal Fatigue with Neural Embeddings got its best results by subtracting a speaker prototype vector. Fisher vectors with cascaded normalization isolated speaker z-normalization as its most important component on eating-condition speech. None of these papers tests the opposite regime, where the label is constant per speaker. We test both regimes on purpose.

The warning about evaluation comes from the IEMOCAP reality check. It showed that apparent gains collapse once test speakers are unseen, so every experiment here uses speaker-independent folds.

The method comes from component-contribution analysis. From Blindfold Baselines for Embodied QA, where a question-only model beat navigation agents, we take the idea of a metadata-only baseline. From DAIC-WOZ: On the Validity of Using the Therapist's prompts, which showed that text models score F1 0.88 from interviewer prompts alone, we take the habit of asking what shortcut a gain rides on. From the LLM-for-time-series deconstructions (Are Language Models Actually Useful for Time Series Forecasting?, Understanding Why Large Language Models Can Be Ineffective in Time Series Analysis) we take the practice of replacing a component with a matched trivial substitute.

Our work differs from all of these in its object of study. We ablate a preprocessing statistic, not a network module or an input modality. A preprocessing statistic is a component whose contribution can leak label information through the reference set. The combinatorial explosion that usually makes exact attribution impractical does not arise here. With six feature families and a linear classifier, all 64 subsets are cheap to evaluate.

Research Program

The primary testbed is the SH2 smartphone corpus (887 speakers, 28 manufacturers, six tasks, depression defined as PHQ-9 of 10 or more). We use the openSMILE feature set of the original study so that its numbers are the reference. We group the low-level descriptors into six physically motivated families: pitch level and variability, loudness, low-order cepstra including the aspiration-sensitive second coefficient, higher-order cepstra, voice quality (jitter, shimmer, harmonics-to-noise), and timing.

For every family we define a set of normalization variants:

  • no normalization;
  • gender normalization with all-speaker statistics, which reproduces the original study;
  • gender normalization with healthy-only training-fold statistics, following the equation above;
  • gender normalization with prevalence-balanced statistics;
  • a matched random control that normalizes by randomly assigned pseudo-groups of the same sizes;
  • manufacturer normalization;
  • per-speaker normalization computed across the six tasks, which keeps each task's deviation from the speaker's own mean.

The random-group control plays the role of the "LLM replaced by a linear layer" substitute. It tells us how much of any gain comes from rescaling as such rather than from group structure.

On top of these variants we compute exact Shapley values. Each family's contribution to depressed-class F1 is measured by switching its normalization from none to a given variant, averaged over all on/off configurations of the other families. The resulting signed attributions define a selective policy that normalizes only the families with positive attribution. The policy is chosen on inner folds only.

We run the same protocol on two corpora where the label varies within speakers. The first is the INTERSPEECH 2015 eating-condition challenge, where 30 speakers each produce every class. The second is IEMOCAP under ten-fold leave-one-speaker-out. The regime prediction is testable in the sign of the per-speaker variant's attribution. It should be positive on these two corpora and negative for the pitch-variability and low-cepstrum families on SH2. As a replication in the constant-label regime, we will apply the pipeline to DAIC-WOZ participant audio if it is accessible. We will also add a blindfold that uses only the interviewer's audio, which fills the acoustic gap the DAIC-WOZ prompt study explicitly left open.

Baselines

The naive reference point is the un-normalized linear SVM on SH2 functionals. Its reported depressed-class F1 is 0.290 (Depression Detection from Short Utterances).

The strongest existing single-model method is that paper's gender-specific normalization at F1 0.396. The strongest system overall is its logistic-regression fusion of task-specific models at depressed F1 0.422, healthy F1 0.823, and 72.9% accuracy. We will re-run both on identical speaker-independent folds, with the same classifier family and the same hyperparameter search budget as our variants. Reported and reproduced numbers will be shown side by side. Beating the single-model baseline means that the healthy-reference selective policy exceeds the reproduced gender-normalized F1 across folds, with a bootstrap confidence interval over speakers that excludes zero difference. Beating the fused system means that the selective policy, dropped into the same task-fusion pipeline, exceeds its reproduced score.

Two further baselines target the claim rather than the leaderboard. The blindfold baseline predicts depression from gender, manufacturer and task metadata alone. It bounds how much any gain could come from demographic shortcuts. The gender-exclusive modeling baseline trains separate un-normalized classifiers per gender. It is reported to help in suicide-risk assessment (Exploring Gender-Specific Speech Patterns), which reports 81% balanced accuracy on a different 20-patient corpus; that is not comparable to SH2.

The key ablation compares healthy-reference and all-speaker statistics while holding everything else fixed. That comparison isolates the contamination mechanism.

On the within-speaker corpora, the reference numbers are the cascaded Fisher-vector system at 83.1% test UAR on eating condition and Cai et al.'s wav2vec 2.0 model at 78.15% weighted accuracy under speaker-independent IEMOCAP. We use these to anchor where the feature pipelines sit, not as targets to beat. For DAIC-WOZ acoustic models, no reported result was found on Lacuna.

Why not just…?

Why not just fine-tune a self-supervised speech model and let it learn invariance? Evidence in the file says such models do not remove identity on their own. The IEMOCAP audit shows accuracy dropping when speakers are unseen. The fatigue work needed explicit prototype subtraction before embeddings separated fresh from fatigued speech. More importantly, the contamination question applies equally to embeddings. Any normalization of embedding features by group statistics inherits the $p_g\delta$ term. We will therefore repeat the healthy-versus-all-statistics comparison on early-layer wav2vec 2.0 embeddings, which the fatigue study found retain voice quality. If an un-normalized fine-tuned model matches the selective policy on SH2, the practical claim weakens. The attribution finding would still stand, because it concerns what the normalization step does.

Why not just read feature importance from LASSO weights or SHAP on a normalized model? Those tools rank features within one fixed preprocessing choice. They cannot say whether normalizing a feature helped or hurt, because the counterfactual un-normalized feature is never in the model. As the NLP analysis survey notes, weights can mislead where erasure does not. We will still report LASSO-derived rankings. If they reproduce the sign pattern of our Shapley attributions, the cheaper tool suffices and we will say so.

Why not just use gender-exclusive models or add gender as a covariate? This is the most serious threat. Stratified models remove the anatomical offset without any shared normalization statistics, so they partly sidestep the contamination. They are included as a full baseline under the same budget. The distinguishing prediction concerns data efficiency: stratified models halve the training set per model, while healthy-reference normalization does not. We will therefore compare the two across training-set fractions. If stratification matches the healthy-reference policy at all sizes, the recommendation becomes "stratify," and the proposal's contribution narrows to the diagnosis of why all-speaker normalization underperforms.

Proposed Experiments and Decisive Checks

The first decisive check is the prevalence test on SH2. If PHQ-9 positive rates differ by gender and healthy-reference normalization beats all-speaker normalization, contamination is real. If the two tie, the gain is anatomical and the equation above is empirically negligible. Either outcome is informative.

The second check is the sign of per-family attributions for per-speaker normalization. We predict it is negative for pitch variability and low-order cepstra on SH2 and positive for the same families on eating condition and IEMOCAP. A uniformly positive pattern on SH2 would refute the erasure hypothesis for depression.

The third check is the matched random-group control. If it recovers most of the 0.290 to 0.396 gain, the published improvement reflects rescaling rather than gender.

The fourth check is the blindfold. If metadata alone approaches the un-normalized acoustic baseline, every SH2 result must be read against that floor.

Metrics are depressed-class F1 and unweighted average recall on SH2, UAR on eating condition, and weighted and unweighted accuracy on IEMOCAP. All are computed under speaker-independent folds with speaker-level bootstrap intervals.

The main risk is access to SH2. If it is unavailable, DAIC-WOZ audio becomes the primary constant-label corpus, at the cost of losing a reported baseline.