Modern neural speech enhancers, the networks that take a noisy recording and output a cleaned one, are accurate and opaque. When one of them muffles a vowel or turns a whistling fan into a warbling artifact, nobody can say which internal decision went wrong, and nobody can correct it short of retraining. Image classification has partly escaped this problem with concept bottleneck models. These are networks forced to predict human-readable quantities first and then compute their answer only from those quantities, which lets a person inspect and overwrite the quantities at test time. Speech has an unusually good set of candidate quantities already defined by physics. Voiced speech is quasi-periodic: it repeats at the pitch of the vocal folds, its energy sits on a comb of harmonics at multiples of that pitch, and most environmental noise does not share this structure.
We propose an enhancer whose cleaning mask is built from a small set of frame-level physical concepts. Those concepts are pitch, voicing probability, a harmonic gain envelope and aperiodic band gains. Anything the concepts cannot express must pass through an explicitly budgeted residual channel, and the model reports how much it used that channel. The concept labels come for free, because enhancement training data already pairs every noisy clip with its clean source, and pitch and harmonic-versus-aperiodic decompositions can be computed from the clean signal. The question is quantitative. How much of a strong black-box enhancer's quality can be routed through physically meaningful, editable quantities? Does editing those quantities at test time, for example fixing an octave error in pitch, repair the output the way concept interventions repair predictions in vision?
Core Claim
A single-channel enhancer can be made faithful by construction. Its output is a known, shallow function of predicted pitch and harmonicity concepts plus a residual whose share of output energy is bounded and measured. We expect this design to sit near the quality of unconstrained models on VoiceBank+DEMAND while supporting two things those models cannot. The first is auditing: what fraction of the cleaned speech is explained by the concepts, frame by frame. The second is intervention: correcting a concept changes the output in a predictable direction. The new element is not "add pitch to an enhancer." It is the combination of three properties. The mask is algebraically tied to the concepts. The concept supervision is derived automatically from the clean reference. Leakage around the bottleneck is treated as a measured, tunable quantity rather than an uncontrolled side channel.
Relation to Prior Work
Three single-channel lines come closest, and each lacks one of these properties. TridentSE routes global context through 32 learned tokens and reaches PESQ 3.47 on VoiceBank+DEMAND. It reports that some tokens appear to track harmonics and noise bands, but this specialization is emergent and unenforced, and the tokens cannot be edited. The paper motivates our work, since it shows that a narrow bottleneck need not hurt quality, and it is a direct competitor. Explaining Deep Learning Models for Speech Enhancement applies DeepSHAP after training and finds that masking networks whose attributions fall inside speech regions give lower downstream word error rate. This supports the premise that "attending to speech structure" predicts good behavior. However, the explanation is post hoc, expensive, and offers no handle for correction. PGSS conditions a separation generator on explicit pitch contours. Pitch is trained separately and the generator remains opaque, so nothing guarantees the output depends on pitch the way a reader assumes. That gap is exactly the one we close.
The multichannel Explainable DNN-based Beamformer with Postfilter obtains interpretability by forcing time-invariant spatial weights whose beampatterns can be plotted. Its single-channel postfilter stays a black box. Its beampattern analysis also exposed a real failure: the beamformer pointed at the speaker but did not null interferers. We take two lessons from it. A structural constraint can be both explainable and competitive. And an inspectable structure is valuable precisely because it reveals failures like this one.
The concept vocabulary comes from classical structural enhancement. Comb filter decomposition for robust ASR keeps energy on harmonic teeth and discards the valleys between them, cutting TIMIT error at 5 dB from 76.6% with standard cepstral features to 29.0%. Covariation and weighting of harmonically decomposed streams for ASR splits speech into periodic and aperiodic streams. This split is what lets us keep fricatives, which purely periodic filtering strips away. Single-Channel Speech Enhancement Using Double Spectrum shows that pitch-synchronous structure improves PESQ by about 0.3 over a classical spectral estimator, with only modest intelligibility gains. That result constrains us: rigid harmonic rules alone leave quality on the table, which is why a budgeted residual exists. MF-PAM supplies a compact, noise-robust pitch estimator built from periodic convolutions, which we reuse as the pitch head.
On the interpretability side, Concept Bottleneck Models supplies three things: the training regimes (independent, sequential, joint), the intervention protocol, and the evidence that bottlenecks can cost little accuracy. That evidence is 19.9% versus 17.5% error on CUB-200-2011, and lower error than the black box on knee X-rays. TesNet shows that orthogonality constraints on concepts can act as a regularizer. Sum-of-Parts Models proves that attribution to single features fails when features are strongly correlated, and that attributing to groups fixes this. Spectrogram bins along a harmonic track are exactly such a correlated group, so we attribute to harmonic combs rather than to individual time-frequency bins. All of this vision work assumes discrete classification. Our setting is a continuous regression output over a time-frequency grid, where "the answer is a function of the concepts" has to be engineered into the mask itself.
Research Program
The model has an encoder: a conformer over the noisy short-time Fourier transform, sized to match published enhancers at roughly 1–2M parameters. For each frame $t$, it outputs a pitch $f_{0,t}$, a voicing probability $v_t$, a harmonic gain envelope $H_t$ over $B$ auditory bands, and aperiodic band gains $g_t$ over the same bands. These are about $2B+2$ numbers per frame. A fixed differentiable renderer turns them into a magnitude mask:
$$M_c(t,f) = v_t\, H_t(b(f)) \sum_{k} \phi_\sigma\!\left(f - k f_{0,t}\right) + g_t(b(f)).$$
Here $b(f)$ maps a frequency bin to its band, and $\phi_\sigma$ is a narrow tooth kernel of width $\sigma$, so that harmonic $k$ passes energy near $k f_{0,t}$. The enhanced spectrum is $\hat S = (M_c + r)\,Y$, where $Y$ is the noisy spectrum and $r(t,f)$ is a residual mask produced by a small separate head. Phase is a known weak point of magnitude masking, as MP-SENet documents with its "compensation effect". We therefore handle phase with a small refinement head, and we count its contribution as part of the residual for auditing purposes.
Training minimizes
$$\mathcal{L} = \mathcal{L}_{\text{SE}}(\hat S, S) + \lambda\, \mathcal{L}_{\text{concept}} + \beta\, \lVert r\,Y \rVert^2 .$$
$\mathcal{L}_{\text{SE}}$ is a standard magnitude and complex-spectrum loss against the clean spectrum $S$. $\mathcal{L}_{\text{concept}}$ supervises the concepts against targets computed from $S$. Pitch and voicing targets come from a pitch tracker run on clean speech. Harmonic and aperiodic gain targets come from a pitch-scaled periodic/aperiodic decomposition of clean speech, divided by the noisy band energy. The weight $\beta$ sets the price of bypassing the concepts. Sweeping $\beta$ traces an explicit accuracy-interpretability curve. The audited quantity is the concept-explained energy fraction: the share of $\lVert \hat S\rVert^2$ attributable to $M_c Y$, reported per utterance and per frame. We compare joint, sequential and independent training as in the original concept bottleneck paper. Sequential training is the variant where the decoder only ever sees predicted concepts, and it should leak least.
Baselines
The naive reference point is the unprocessed noisy input, together with a classical structural estimator. We will reimplement the Double Spectrum weighting of Single-Channel Speech Enhancement Using Double Spectrum and also run a comb-filter mask driven by the same pitch tracker. Both are fully transparent and training-free. Any learned concept model that fails to beat them clearly has no reason to exist. That paper reports an average 0.3 PESQ gain over a classical spectral estimator on Noizeus babble, and our reimplementation will be checked against that figure.
The strongest existing method is MP-SENet, which reports PESQ 3.50 on VoiceBank+DEMAND with 2.05M parameters (MP-SENet). TridentSE reports 3.47 there and 3.44 on the DNS no-reverb test set (TridentSE). We will retrain both from public descriptions on identical data, with matched parameter count (within 10%), the same number of training steps, and the same hyperparameter search budget as our model. We report our retrained numbers alongside the published ones. Beating them outright is not the claim. The claim succeeds if, at a residual budget where at least 90% of output energy is concept-explained, our model is within 0.1 PESQ of retrained MP-SENet and within one STOI point. This is a pre-registered target, not a reported figure.
The baseline most threatening to the idea is a pitch-conditioned black box. This is MP-SENet with the same predicted concept tracks concatenated to its input, in the spirit of PGSS. It sees the same information and is equally "pitch-aware," but nothing forces it to use pitch. Two ablations isolate our key ingredient. The first replaces the physical concepts with an unsupervised latent bottleneck of the same width ($2B+2$ numbers per frame), pushed through a learned rather than fixed renderer. This tests whether physical grounding matters beyond narrowness, and it is a close cousin of TridentSE's tokens. The second sets $\beta = 0$, an unlimited residual, to show how quickly the model abandons the concepts when bypassing them is free.
Why not just…?
Why not just explain MP-SENet with DeepSHAP? Post-hoc attribution reports which bins mattered. It does not ensure that the model's computation runs through speech structure, and it gives the user no handle to change the result. Explaining Deep Learning Models for Speech Enhancement had to collapse its attributions across time or frequency to make them readable, and Sum-of-Parts Models shows why per-bin attributions are unreliable when bins are as correlated as harmonics are. We will still compute the speech relevance score from that paper on our model and on MP-SENet. If MP-SENet scores as high, with comparable cost and the same ability to localize failures, the auditing benefit is weaker than claimed, though the intervention benefit would remain.
Why not just feed pitch into a strong black box? This is the real threat. It may match our quality while looking just as interpretable. The deciding experiment is a faithfulness test. We perturb the pitch input by a known amount, for example forcing octave errors or shifting by a semitone, and measure how much and how predictably the output harmonics move. In our model the harmonic teeth move by construction. If the conditioned black box also responds strongly and monotonically, then the fixed renderer buys little. In that case we would reframe the contribution around the residual budget and the audit metric rather than the renderer.
Why not just use classical comb or Double Spectrum filtering? These methods are transparent, but they are capped by rigid rules. The Double Spectrum method's gains in intelligibility are modest, and the comb-filter features in Comb filter decomposition for robust ASR still leave 29.0% error at 5 dB. Our $\beta \to \infty$ endpoint is essentially a learned version of these filters. Comparing it against the classical ones measures how much of our gain comes from learning to predict concepts under noise, as opposed to learning the residual.
Why not just use TridentSE's tokens and interpret them afterward? Emergent token specialization is not guaranteed, is not stable across training seeds, and is not editable. The latent-bottleneck ablation tests this directly. We will train both it and our model over five seeds and measure two things: how well latent dimensions align with clean-derived pitch and harmonicity, and whether editing a latent dimension produces a consistent acoustic effect across seeds.
Proposed Experiments
Main quality results are reported on VoiceBank+DEMAND (PESQ, STOI, and the composite CSIG/CBAK/COVL scores) and on the DNS no-reverb test set. Noizeus babble at 0, 5 and 10 dB allows a direct comparison with the Double Spectrum results. Concept accuracy is measured as raw pitch accuracy on VCTK-DT, comparing our pitch head with MF-PAM run on noisy input. We will also remix VoiceBank+DEMAND test speech at −5 and −10 dB. This stresses the regime where pitch extraction is known to fail, and we expect the concept-explained fraction to drop there, the residual to take over, and the audit to flag it.
Three checks decide whether the idea holds. The first is the curve of PESQ against concept-explained energy fraction across $\beta$, compared with MP-SENet's single point. The second is intervention gain, following the oracle protocol of the concept bottleneck paper: we replace predicted pitch and voicing with clean-derived values on the worst 10% of frames and measure the PESQ and STOI recovered. We do the same for a simulated user who corrects only octave errors, and we report how many corrected frames are needed to recover a given fraction of the gap to oracle. The third is the perturbation faithfulness test against the pitch-conditioned black box described above.
A secondary check runs the CHiME-4 real test set through a fixed speech recognizer. The reference points are 25.9% word error rate unenhanced and 11.7% for the best enhancer reported in Explaining Deep Learning Models for Speech Enhancement. We ask whether frames with high concept-explained fraction carry fewer recognition errors, which would make the audit signal practically useful.
The idea fails if any of three things happens. Quality at 90% concept-explained energy falls more than 0.2 PESQ below retrained MP-SENet. Oracle interventions recover little. Or the pitch-conditioned black box matches our faithfulness.
References
- TridentSE: Guiding Speech Enhancement with 32 Global Tokens
- Explaining Deep Learning Models for Speech Enhancement
- PGSS: Pitch-Guided Speech Separation
- Explainable DNN-Based Beamformer With Postfilter
- Comb filter decomposition for robust ASR
- Covariation and weighting of harmonically decomposed streams for ASR
- Single-Channel Speech Enhancement Using Double Spectrum
- MF-PAM: Accurate Pitch Estimation through Periodicity Analysis and Multi-level Feature Fusion
- Concept Bottleneck Models
- Interpretable Image Recognition by Constructing Transparent Embedding Space
- Sum-of-Parts Models: Faithful Attributions for Groups of Features
- MP-SENet: A Speech Enhancement Model with Parallel Denoising of Magnitude and Phase Spectra