Speech enhancement models (networks that remove background noise from recorded speech) are increasingly trained on real recordings rather than synthetic mixtures. Since no clean reference exists for a real recording, the training target has to be a "pseudo-label": an approximate clean signal, often taken from a close-talk microphone worn by the speaker. These pseudo-labels have errors, and the errors are not spread evenly. They should be worst on unvoiced consonants such as /s/, /f/, /t/. These sounds are quiet and noise-like, and a cleaning network is likely to erase them along with the noise. A student model trained on such targets learns to delete consonants, and the deletion carries over to every new dataset it is applied to.
We propose to stop treating the pseudo-label as a single point target in the places where it is least trustworthy. Where an independent phonetic detector says an unvoiced consonant is present, and the cleaning step has removed high-frequency energy, the target becomes an interval. Its lower bound is the cleaned close-talk signal, which is probably over-suppressed. Its upper bound is the raw close-talk signal, which is probably under-suppressed because it still contains leaked noise. The student is penalized only when its output falls outside that range. The claim is that this asymmetric, phonetically localized treatment recovers consonants that point-target pseudo-labeling suppresses. We also claim the gain holds up under cross-corpus evaluation on phonetically labeled English and German speech, which is where the consonant loss is actually measurable.
What exists and what is missing
The closest work is ctPuLSE. It trains a far-field enhancement model on real CHiME-4 recordings using close-talk pseudo-labels. The close-talk signal is first cleaned by a network trained on simulated data (CTSEnet), then aligned to the far-field output with a learnable linear filter, and training mixes in simulated data with true targets. It improves real-data DNSMOS from 1.39 unprocessed and 3.21 for a simulation-only model to 3.29, and cuts monaural recognition error from 4.47% to 3.13%. Every time-frequency bin gets the same point-target loss, and the system has no mechanism for deciding where the cleaned target is wrong. Our method starts from this pipeline and changes only the target in consonant regions.
Two vision papers motivate the idea that pseudo-label noise clusters in predictable places. Combating Noise finds that detection pseudo-label errors concentrate near the decision threshold and softens targets there. Auto-Annotation Quality Prediction masks the 3.8% least reliable pixels and reaches 77.0% mIoU with 30% of the labels, matching full supervision. Both estimate reliability from the teacher's own confidence or from ensemble agreement. That is exactly the kind of signal that fails when the teacher is systematically and confidently wrong about a whole class of sounds, and the research file flags this teacher-bias ceiling as an open problem. Our reliability signal is deliberately external to the teacher.
The unvoiced-speech literature supplies the acoustic mechanism. HGCN compensates only voiced harmonic regions and leaves unvoiced regions to a coarse first pass. Hu et al. show that unvoiced speech sits in the residue after periodic components are removed, and that it concentrates above about 2 kHz. DNN-Struct shows that temporal context raises the rate of correctly kept unvoiced bins (hits minus false alarms) from 41.0% to 65.0%, because isolated unvoiced frames look like noise. Yurt et al. provide a low-latency raw-waveform fricative detector with 92.79% unweighted average recall on TIMIT, but they test it only on TIMIT and on coded or bandlimited versions of TIMIT.
Two papers constrain the design. Corpus-testing a fricative discriminator shows that a spectral-slope rule for /s/ versus /ʃ/ keeps roughly 91–93% accuracy on /s/ across TIMIT and German Kiel corpora. However, its best threshold moves from about 2 to 2.5 dB/kHz between corpora, compared with 8–15 dB/kHz in earlier literature. Any fixed acoustic threshold is therefore fragile across corpora, and our detector has to be validated across corpora before it can be trusted as a reliability signal. Complex spectral mapping shows that more diverse training data and a smaller frame shift improve cross-corpus transfer. This is the main reason to worry that data diversity alone would fix the problem.
Core Claim
In close-talk pseudo-labeling, errors in unvoiced consonant regions are systematically biased toward over-suppression. An interval target in those regions, bounded by the cleaned and raw close-talk signals and located by an independent phonetic detector, removes that bias without discarding the supervision. Formally, let $\hat S$ be the student's magnitude spectrogram, $L$ the aligned cleaned close-talk magnitude, $U$ the aligned raw close-talk magnitude, and $m\in[0,1]$ the reliability flag per time-frequency bin. The loss on real data is
$$ \mathcal{L} = \sum_{t,f} (1-m_{t,f})\,\ell\big(\hat S_{t,f}, L_{t,f}\big) + m_{t,f}\Big(\big[L_{t,f}-\hat S_{t,f}\big]_+ + \big[\hat S_{t,f}-U_{t,f}\big]_+\Big), $$
where $\ell$ is ctPuLSE's usual point loss and $[\cdot]_+$ is the positive part. Here $m_{t,f}$ is the product of two quantities. The first is the detector's unvoiced probability for frame $t$, computed on the raw close-talk channel, which has high signal-to-noise ratio. The second is an indicator that cleaning removed more than a set number of decibels in bin $(t,f)$ above 2 kHz. Everywhere else, training is identical to ctPuLSE.
Research Program
The first stage tests the premise before building anything. We simulate close-talk conditions by adding low-level CHiME-4 background noise to clean speech from TIMIT, which has phone alignments. We run CTSEnet on the result and break the pseudo-label error down by phone class and by direction of error. If the error in unvoiced regions is neither disproportionately large nor biased toward suppression, the proposal's premise is false, and we would report that rather than proceed.
The second stage builds and cross-validates the detector. We train a detector in the style of Yurt et al. on TIMIT and evaluate it on Kiel read and spontaneous speech, on LibriSpeech with forced alignments, and on CHiME-4 close-talk audio with forced alignments. We report the recall lost at each step. This is also a contribution in its own right, because it is the cross-corpus test that Yurt et al. never ran.
The third stage trains ctPuLSE-style models on CHiME-4 real and simulated data under the variants listed in the Baselines section. The fourth stage evaluates all models on corpora none of them was trained on.
Baselines
The do-nothing reference is the unprocessed far-field signal, together with a TF-GridNet model trained only on simulated data. ctPuLSE reports DNSMOS values of 1.39 and 3.21 for these two on real CHiME-4 data.
The strongest existing method is ctPuLSE itself. It reports DNSMOS 3.29 and monaural WER 3.13% on the real CHiME-4 test set, and 1.56% WER with six channels. We will retrain it with the same TF-GridNet backbone, the same data mix, the same number of update steps, and the same tuning budget as our model. Beating it means two things. First, a statistically significant gain in fricative-segment recovery on the held-out phonetically labeled corpora. Second, no loss on CHiME-4 real DNSMOS or WER. A consonant gain that costs overall quality does not count.
A confidence-filtering baseline adapts Combating Noise. It down-weights bins where an ensemble of three differently seeded CTSEnet models disagrees, with the total loss weight kept equal to ours. This is the generic pseudo-label-noise remedy, and it isolates the question of whether an external signal beats one derived from the teacher.
A supervised diversity baseline follows complex spectral mapping. It trains on simulated LibriSpeech mixtures with a quarter-frame shift and no real data.
Three ablations isolate the key ingredients. The first keeps the detector flags but replaces the interval with a zero weight ("ignore"), which tests whether the interval matters beyond simply masking bins. The second drops the detector and flags every high-frequency bin where cleaning removed energy, which tests whether phonetic localization matters. The third applies a global high-frequency loss weight with matched total weight.
No reported result was found on Lacuna for any of these methods on consonant-specific metrics or on cross-corpus transfer from CHiME-4.
Why not just…?
Why not just up-weight high frequencies in the loss? Unvoiced energy lives mostly above 2 kHz, so a global frequency weighting might recover much of the gain. It would also, however, push the student to keep high-frequency noise such as wind and babble in every frame, which is the confusion the research file identifies. The matched-weight frequency ablation settles this. If it matches our fricative-segment gains without raising residual noise in non-speech frames, the phonetic machinery is unnecessary, and we would say so.
Why not just use teacher confidence or ensemble agreement? When every teacher shares the same bias of treating fricatives as noise, the teacher is confidently wrong and its ensemble members agree with each other. That is the failure mode raised against Auto-Annotation Quality Prediction-style filtering. The confidence-filtering baseline tests this directly. If it performs as well as our method, the claim that the reliability signal must be external is wrong.
Why not rely on simulated co-learning, or on diverse supervised data? ctPuLSE already mixes in simulated data with perfect targets, and diversity improves transfer according to the cross-corpus study. This is the most serious threat. If the simulated half of training already anchors consonants, the premise check in the first stage will show little bias on real data, and the LibriSpeech baseline will match us on held-out corpora. Either outcome would falsify the proposal.
Why not skip the detector and flag any bin where cleaning removed high-frequency energy? This is cheap, needs no labeled phonetic data, and avoids the detector's own cross-corpus fragility. It is the second ablation. If it matches the full method, we would recommend it instead.
Proposed Experiments
Evaluation uses six sets. On CHiME-4 real, we report DNSMOS and WER, plus deletion rates for fricatives and stops from forced alignment against transcripts. On CHiME-4 simulated, we report SI-SDR (scale-invariant signal-to-distortion ratio) and PESQ (a standard objective speech-quality score). The main cross-corpus tests are TIMIT and IEEE sentences, and German Kiel read and spontaneous speech, mixed with unseen noises at −5 to 0 dB. On these we measure segmental SNR and high-band energy ratio restricted to labeled fricative and stop segments, alongside whole-utterance SI-SDR and STOI (a standard objective intelligibility score). Finally, we evaluate the INTERSPEECH 2020 DNS real recordings with ITU-T P.835 listening tests, which separate speech distortion from noise removal, because P.808 cannot make that distinction (DNS Challenge).
The decisive check combines three results. The first stage must show that unvoiced pseudo-label error is biased toward suppression. Our model must then beat ctPuLSE on fricative-segment recovery on Kiel and TIMIT with no global regression. Finally, both the zero-weight and the detector-free ablations must fall measurably short, since only that isolates the interval target and the phonetic localization as the reasons for the gain.
References
- ctPuLSE: Close-talk, and pseudo-label based far-field, speech enhancement
- Combating Noise: Semi-supervised Learning by Region Uncertainty Quantification
- Auto-Annotation Quality Prediction for Semi-Supervised Learning with Ensembles
- HGCN: harmonic gated compensation network for speech enhancement
- Unvoiced speech segregation based on CASA and spectral subtraction
- Boosting Classification Based Speech Separation Using Temporal Dynamics
- Fricative Phoneme Detection Using Deep Neural Networks and its Comparison to Traditional Methods
- Corpus-testing a fricative discriminator; or, just how invariant is this invariant?
- Learning Complex Spectral Mapping for Speech Enhancement with Improved Cross-Corpus Generalization
- The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets,\n Subjective Testing Framework, and Challenge Results