Emotion recognizers tell anger from joy largely through voice quality, meaning how breathy, creaky or tense the voice sounds. Voice quality is read off small differences between harmonic peaks in the spectrum, and those peaks are the first thing background noise buries. The common fix is to denoise the audio first, but denoising smooths away exactly those cues. This proposal borrows a lesson from polarization imaging, where a faint quantity is computed as a ratio of two small numbers. That ratio does not merely get noisy at low signal. It becomes meaningless, so it is used only together with a confidence value that says how much signal was present. We propose to treat each voice-quality measure the same way: as a noisy measurement to be corrected for noise bias, paired with a per-frame confidence, and fed to the emotion model through a gate that trusts it only where the confidence is high. The raw waveform is never altered.
The target is a specific, documented failure. The adaptive-enhancement system NRSER improves valence and categorical emotion scores in noise, but its authors report that it loses effectiveness on arousal and dominance. They attribute this to enhancement removing micro-prosodic and voice-quality cues (Noise robust speech emotion recognition with signal-to-noise ratio adapting speech enhancement). If voice quality can be recovered from noisy speech without enhancement, arousal and dominance in noise are where the gain should show up. They are also where the idea is falsifiable.
Core Claim
Harmonic voice-quality measures behave in noise like the angle of polarization. The relevant measures are $H1-H2$ (the level gap between the first two harmonics, which tracks breathiness), $H1-A3$ (first harmonic against the third formant peak, which tracks how sharply the vocal folds close), and cepstral peak prominence (a periodicity measure). Each is a ratio of spectral quantities that collapses into noise-floor fluctuation as local signal drops. In polarimetry, the angle $\frac{1}{2}\arctan(U/Q)$ becomes uniform noise as the degree of polarization goes to zero. The car detector PCDNet handles this by using the degree of polarization as a confidence weight on the angle (Exploiting Polarized Material Cues for Robust Car Detection). The speech literature does the opposite. It either computes tilt only on frames a pitch tracker labels voiced, which is brittle in noise, or discards handcrafted voice quality in favor of raw neural embeddings.
Our claim has three parts. Debiasing and confidence-weighting these measures preserves their information down to lower signal-to-noise ratios (SNRs) than the roughly 10 dB where they are known to collapse (Evaluation of Spectral Tilt Measures for Sentence Prominence Under Different Noise Conditions). Adding them through a confidence gate to a strong self-supervised speech encoder recovers arousal and dominance accuracy that both enhancement and noise-augmented fine-tuning fail to recover. And the gate, not the features alone, is what makes this work.
Research Program
The first component is a measurement layer that works on the noisy waveform directly. For each frame $t$ and harmonic $k$, we estimate the noise power $\hat N_{t,k}$ near that harmonic's frequency from inter-harmonic valleys and minimum-statistics tracking. We then subtract it from the observed harmonic power $|X_{t,k}|^2$ before taking any ratio:
$$\hat P_{t,k} = \max\!\left(|X_{t,k}|^2 - \hat N_{t,k},\ \epsilon\right), \qquad c_{t,k} = \sigma\!\left(\alpha \log\frac{|X_{t,k}|^2}{\hat N_{t,k}} - \beta\right).$$
Here $\hat P_{t,k}$ is the debiased harmonic power and $\epsilon$ is a small floor. The confidence $c_{t,k}$ is a squashed local harmonic-to-noise ratio, with $\sigma$ the logistic function and $\alpha, \beta$ learned scalars. The subtraction matters because additive noise inflates weak harmonics more than strong ones. Without it, $H1-H2$ is pulled toward zero as SNR drops, and the model reads noise as a change in phonation.
Uncertainty in pitch is propagated rather than thresholded. The harmonic positions are taken from a posterior over fundamental frequency, and frames with diffuse posteriors receive low confidence instead of being excluded. This removes the dependence on a hard voiced/unvoiced decision that limited the spectral-tilt study cited above. Each voice-quality measure $v_t$ inherits a confidence $c_t$ from its constituent harmonics, for example the minimum of the two confidences involved in $H1-H2$.
The measurement layer also follows the finding that emotional information in speech clusters below 300 Hz and around the third formant near 3 kHz, while the first two formants mostly carry vowel identity (Physiologically-inspired feature extraction for emotion recognition). Confidence is therefore computed separately in these two bands. Babble noise that masks the low band but spares 3 kHz should then downweight $H1-H2$ while keeping $H1-A3$.
The second component is the fusion. A frozen, pre-trained HuBERT encoder supplies a weighted mix of its layers. Prior work showed that intermediate layers carry the most affective information and that fine-tuning for speech recognition strips it out (Emotion Recognition from Speech Using wav2vec 2.0 Embeddings). A small cross-attention module lets the encoder's frame embeddings query the voice-quality stream, with attention logits offset by $\log c_t$ so that low-confidence frames contribute almost nothing. This is the speech analogue of PCDNet's query from the reliable stream into the unreliable one.
Utterance-level pooling of voice quality is confidence-weighted:
$$\bar v = \frac{\sum_t c_t v_t}{\sum_t c_t}.$$
This pooled summary is used for the interpretability analyses described below.
The design does not use contact sensors, which get noise immunity at the cost of practicality and vocal-tract information (Electroglottograph-Based Speech Emotion Recognition via Cross-Modal Distillation). It also does not use an enhancement front end, which has been shown to damage even articulatory estimation compared with leaving noise in place (Noise Robust Acoustic to Articulatory Speech Inversion).
Baselines
The naive reference is HuBERT with a weighted-layer emotion head, trained on clean MSP-Podcast and tested on noisy mixtures. It shows how much noise costs when nothing is done.
The strongest existing method on this exact question is NRSER (Noise robust speech emotion recognition with signal-to-noise ratio adapting speech enhancement). NRSER runs CMGAN enhancement, measures the spectrogram cosine similarity between input and enhanced audio, and uses that score to blend the two waveforms before a HuBERT backend. Its reported quantitative result in the file is above 90% accuracy at detecting pure-noise segments at a similarity threshold of 0.6. No reported per-SNR CCC (concordance correlation coefficient) for arousal, valence or dominance was found on Lacuna for NRSER, because those values appear only in a figure. We will therefore reimplement NRSER using CMGAN trained on Voice Bank+DEMAND, as in the original work, on our identical MSP-Podcast v1.10 split with AudioSet noise.
An equally important baseline, and the one most likely to be hard to beat, is HuBERT fine-tuned end to end with multi-condition noise augmentation over the same SNR range. All systems share the HuBERT checkpoint, training data, noise clips, SNR schedule, number of hyperparameter trials (a fixed budget of 20 each) and GPU hours.
For IEMOCAP, the external reference is the 78.15% weighted accuracy under a 10-fold speaker-independent protocol, as tabulated in Designing and Evaluating Speech Emotion Recognition Systems: A reality check case study with IEMOCAP. We follow that paper's protocol, including its guidance on class grouping, to avoid inflated numbers.
Three ablations isolate the key ingredients. The first concatenates raw, non-debiased voice-quality features without any gate. The second adds debiasing but uses hard frame dropping below a confidence threshold instead of soft gating. The third keeps the gate but replaces our per-band confidence with a single utterance-level SNR estimate.
Beating the baselines means the following. Arousal and dominance CCC at 0–10 dB must exceed both NRSER and the multi-condition HuBERT model, with the difference in average CCC across 0, 5 and 10 dB exceeding the 95% bootstrap interval over test speakers. Clean-condition CCC must stay within that interval of the naive model's clean result, so that no clean-speech penalty is paid.
Why not just…?
Why not just denoise first with a strong enhancer? Two independent results say this removes what we need. NRSER's authors report weaker arousal and dominance gains and blame enhancement for stripping voice-quality cues. Separately, log-MMSE enhancement made articulatory inversion worse than feeding noisy audio to a multi-condition model (Noise Robust Acoustic to Articulatory Speech Inversion). We still run the enhancer as a baseline. We also compute our voice-quality measures on enhanced audio and compare them with the values from the clean reference. That comparison directly tests whether enhancement biases $H1-H2$.
Why not just fine-tune HuBERT with noise augmentation and let it learn voice quality implicitly? This is the real threat. The evidence against it is only indirect: multi-condition training carried a clean-speech penalty and struggled on babble and restaurant noise in articulatory inversion (Noise Robust Acoustic to Articulatory Speech Inversion). It is not direct evidence about emotion. We will settle the question with a probing experiment before building the fusion. Linear probes trained on each HuBERT layer, under noise, will predict the clean-reference $H1-H2$, $H1-A3$ and cepstral peak prominence. If the probes from the noise-augmented model recover these values as accurately as our debiased estimator at 0–10 dB, the explicit measurement layer is redundant. In that case we would pivot to using debiased voice quality only as an auxiliary training target.
Why not just append an SNR estimate, or the raw features, as extra inputs? A classifier given raw $H1-H2$ plus a global SNR must learn, per utterance, a correction for a bias that varies by frame, band and noise type. The first and third ablations above are exactly this alternative. If they match the full model, the bias correction and gating add nothing, and we will report that.
Proposed Experiments
Experiments start at the measurement level, where ground truth exists. We mix clean CGN and MSP-Podcast speech with AudioSet and AURORA-type noise at −5 to 20 dB. We then compare each voice-quality estimate against its value on the clean signal, with and without debiasing, and with hard versus soft confidence. On CGN we repeat the prominence-separation analysis of Evaluation of Spectral Tilt Measures for Sentence Prominence Under Different Noise Conditions. That paper reports clean effect sizes of about 0.42–0.43 for the best tilt measures and collapse around 10 dB. The first decisive check is whether debiased, confidence-weighted tilt keeps its effect size below 10 dB, specifically at 5 dB.
The main experiment is MSP-Podcast v1.10 with AudioSet noise at 0, 5, 10, 15 and 20 dB plus clean speech. We report arousal, valence and dominance CCC and categorical macro-F1 for all baselines and ablations, broken down by stationary versus babble noise. IEMOCAP under the speaker-independent protocol serves as a transfer check on a second corpus. Training uses MSP-Podcast only, and noise is added at test time.
The second decisive check is the pattern of gains. If the mechanism is right, gains should concentrate on arousal and dominance and grow as SNR falls, while valence changes little. Across utterances, the gain should correlate with how far the raw voice-quality features were biased away from their clean values.
The third decisive check is the HuBERT probing study described above. It determines whether the explicit measurement path is necessary at all.
References
- Noise robust speech emotion recognition with signal-to-noise ratio adapting speech enhancement
- Exploiting Polarized Material Cues for Robust Car Detection
- Evaluation of Spectral Tilt Measures for Sentence Prominence Under Different Noise Conditions
- Physiologically-inspired feature extraction for emotion recognition
- Emotion Recognition from Speech Using wav2vec 2.0 Embeddings
- Electroglottograph-Based Speech Emotion Recognition via Cross-Modal Distillation
- Noise Robust Acoustic to Articulatory Speech Inversion
- Designing and Evaluating Speech Emotion Recognition Systems: A Reality Check Case Study with IEMOCAP