Spiking neural networks are networks whose units communicate with sparse binary pulses ("spikes"), so their energy cost scales with how often they fire. They are the leading candidates for speech enhancement on hearing aids and other always-on devices. The best spiking denoisers share one weakness. They either clean only the magnitude of the time-frequency spectrogram and reuse the noisy phase, which is what Spiking-FullSubNet does, or they skip the spectrogram entirely and filter the raw waveform, which is what DPSNN does at a PESQ of 2.37 against 2.66 for frequency-domain models. Phase is the timing information of each frequency component. Non-spiking models show it matters a great deal: DCUNet reaches PESQ 3.24 and MP-SENet reaches 3.50 on VoiceBank+DEMAND by estimating it explicitly.
Adding phase outputs to a spiking network is the obvious next step, and it is close to a routine engineering exercise. The harder problem is what that step does to the spike budget. Phase looks like high-entropy noise at every time-frequency point. Any compression pressure that is blind to content will therefore remove it. That pressure could be a uniform penalty on spike counts or a generic information bottleneck, meaning a penalty on how much the internal code says about the input. We propose a different pressure. The network is penalized for how much its internal membrane potentials reveal about the noise signal, a quantity we can measure because the clean and noise components of each training mixture are known. It is not penalized for how much it reveals about the input. Our claim is that this lets a spiking enhancer drop the spikes that carry noise while keeping the ones that carry speech phase. The result should be a better trade-off between quality and spike count than spike-rate penalties or generic bottlenecks give, especially on noise types not seen in training.
Core Claim
In a causal spiking enhancer, learned per-band membrane decay provides the temporal memory. Neurons with learnable decay follow PLIF and adaptive thresholds follow ALIF. Parallel magnitude and phase decoders with an anti-wrapping phase loss provide the phase, following MP-SENet.
Within this network, we minimize a neural estimate of the mutual information between hidden membrane potentials and the noise component. We expect this to reach a given PESQ and SI-SNR with fewer spikes than an $L_1$ spike penalty or a variational bottleneck on $I(X;Z)$. We also expect it to hold up better when the noise distribution shifts. The generic bottleneck should fail in a specific, measurable way: it should suppress phase information first.
The objective is
$$\mathcal{L} = \mathcal{L}_{\text{rec}}(\hat{S}, S) + \beta\, \hat{I}_\phi(V; N) + \gamma\, \bar{r},$$
where $\hat{S}$ and $S$ are the estimated and clean complex spectra, $\mathcal{L}_{\text{rec}}$ combines magnitude, anti-wrapping phase and time-domain SI-SNR losses, $V$ is a hidden layer's membrane-potential trajectory over a short causal window, and $N$ is the noise spectrogram over the same window. $\hat{I}_\phi$ is a contrastive estimate from a critic network $\phi$, $\bar{r}$ is the mean firing rate, and $\beta$ and $\gamma$ weight the two penalties. We keep a small $\gamma$ so that every comparison is made at matched spike budgets. The claim concerns where the spikes go, not whether spikes are penalized at all.
The information-bottleneck literature supplies the motivation. Supervised bottlenecks remove specific nuisance content when told what it is: Yao et al. removed calibration markers from ultrasound models this way. For spiking networks, VDIB showed that information-theoretic training of spike codes far outperforms handcrafted decoding, reaching 92.82% against 60.50% on MNIST. That work used a fixed spike response model with no learnable decay, an artificial-network decoder, and no audio. We differ on all three counts. We also replace its compression term, which penalizes information about the input, with a penalty on information about the noise.
Research Program
We first build the substrate: a causal spiking encoder over STFT frames with non-uniform frequency bands in the style of Spiking-FullSubNet. PLIF decay is shared within each band so that bands can learn different memory lengths. ALIF recurrence sits at the bottleneck. A spiking magnitude-mask decoder and a phase decoder with an Arctan2 output head produce the complex spectrum. Following Choi et al., who showed that spectral MSE alone makes the phase correction collapse, the reconstruction loss always includes a time-domain term.
Second, we attach the critic to membrane potentials rather than spikes. This addresses the gradient problem that discrete spikes pose for estimators like MINE. Membrane potentials are continuous and depend differentiably on the weights and on the learnable decay. The critic's gradient therefore reaches $\tau$ directly, and surrogate gradients are needed only for the spike nonlinearity downstream. We use a multi-sample contrastive bound of the kind studied by Xu et al. rather than the Donsker-Varadhan bound, because the log-mean-exponential term in the latter has the variance problems described in the MINE line of work. Negatives are noise windows drawn from other mixtures in the batch.
Minimizing a lower bound does not guarantee the information is gone. For that reason, every trained model is audited afterwards. We freeze the model, train a new critic and a separate regression probe from scratch, and check whether either can still recover the noise.
Third, we run the analysis the method depends on. For every trained model and layer, we estimate how much the hidden code says about the noise, about the clean magnitude, and about the clean phase derivatives. For phase we use instantaneous frequency and group delay, which are smooth where raw phase is wrapped. This yields per-layer information curves showing where each regularizer removes content. It also tests the specific prediction that a generic bottleneck on $I(X;Z)$ removes phase-derivative information before magnitude information. We also record the learned decay constants per band and ask whether the noise penalty shortens memory in bands dominated by stationary noise. This last question is exploratory, and we will report it either way.
Baselines
The naive reference point is the unprocessed noisy input, evaluated with the same metrics. The magnitude-only spiking reference is a reimplementation of Spiking-FullSubNet's recipe on our encoder, reusing the noisy phase. It shows how much phase modeling alone contributes; the file gives no exact N-DNS SI-SNR or DNSMOS value for it, only that it won the algorithmic track. The time-domain spiking reference is DPSNN, which reports DNSMOS 2.94, STOI 0.93 and PESQ 2.37 on VCTK. We will retrain it on our splits rather than rely on those numbers.
The strongest competitor on our specific question is the same phase-aware spiking network trained with reconstruction loss plus an $L_1$ spike penalty. This is DPSNN's regularizer applied to our architecture. It is the most likely thing to work as well as our method.
All spiking variants share the same architecture, data, number of training steps and $\beta$/$\gamma$ search budget, which is a fixed number of runs per method. We compare them along curves of quality against spike rate, not at single operating points. Beating this baseline means a higher PESQ and SI-SNR at matched spike rate, or equivalently fewer spikes at matched quality, over the operating range, with seed-to-seed variance smaller than the gap.
Two more baselines isolate the key ingredient. The first is a variational bottleneck on $I(X;Z)$ inserted at the same layer, which tests whether it matters that the penalty targets noise. The second is a gradient-reversal adversary that regresses the noise spectrogram from the membrane potentials, which tests whether the contrastive critic matters or any noise-removal pressure would do.
Non-spiking phase-aware models serve as a ceiling, not a target. MP-SENet reports PESQ 3.50 on VoiceBank+DEMAND with 2.05M parameters, DCUNet reports 3.24, and DCCRN has 3.7M parameters and runs in 3.12ms per frame on a laptop CPU. Our goal is to shrink the gap to these models at a small fraction of their operation count, not to exceed them.
Why not just…?
Why not just add complex outputs to an existing spiking denoiser and stop there? This is the most serious threat. The reconstruction loss already requires the output to be free of noise. For output quality at a fixed spike budget, the information penalty may therefore add nothing. We accept this possibility, and the $L_1$ baseline above is exactly this model. The proposal predicts an advantage in two places. The first is spike efficiency, because intermediate spikes that encode noise are wasted energy even if the decoder later discards them. The second is robustness to noise not seen in training, because a code that carries no noise identity has less to misfire on. If the penalized and unpenalized curves coincide on both counts, we will report that noise-targeted information penalties are unnecessary for spiking enhancement. That would still be a useful negative result, and the phase-aware spiking substrate would remain a contribution.
Why not just penalize spike counts with $L_1$? An $L_1$ penalty removes spikes uniformly, regardless of what they carry. The file suggests this is weaker in the related setting of decoding: VDIB's information-theoretic training far outperformed handcrafted rate decoding. That evidence is indirect, since it comes from image classification. The matched-spike-rate comparison, together with the information audit showing which content each penalty removes, settles the question for this task.
Why not just use a standard variational information bottleneck? A bottleneck on $I(X;Z)$ has no way to tell speech phase from noise. Both look unpredictable at individual time-frequency points. Earlier work also shows that bottleneck representations can collapse abruptly as the penalty weight crosses thresholds, as analyzed in Phase Transitions for the Information Bottleneck. We therefore expect it to regress toward magnitude-only behavior. Our per-layer information curves test this directly. If the variational bottleneck preserves phase-derivative information as well as our penalty does, the case for noise targeting is lost.
Why not just use an adversarial noise predictor with gradient reversal? This is simpler and may be nearly equivalent. It is included as an ablation. Our reason to expect a gap is that a regression adversary removes only what is linearly or cheaply decodable, whereas a contrastive critic scores any dependence. The post-hoc audit with fresh critics and probes shows which approach actually leaves less recoverable noise.
Decisive Experiments
We train on VoiceBank+DEMAND and report PESQ, CSIG, CBAK and COVL. On the Intel N-DNS dataset we report SI-SNR, DNSMOS and the challenge energy proxy. On VCTK we compare directly with DPSNN, adding STOI and latency.
The robustness test is cross-corpus: train on VoiceBank+DEMAND and test on N-DNS validation mixtures, whose noise sources differ. We will note that the speech also changes between the two corpora. For every method we sweep the penalty weight to trace a curve of quality against spike rate.
Three results decide the proposal. The first is whether the noise-targeted penalty sits above the $L_1$ curve, especially in the cross-corpus setting. The second is whether the fresh-critic audit confirms that less noise information remains, so that the gain is not an artifact of a critic the model learned to fool. The third is whether the variational bottleneck loses phase-derivative information before magnitude information, as the phase-collapse hypothesis predicts.
A positive result on all three would show that phase-aware spiking enhancement can be made spike-efficient by targeting noise. A null result on the first, with positive results on the other two, would show that the reconstruction loss already does this job.
References
- Towards Ultra-Low-Power Neuromorphic Speech Enhancement with Spiking-FullSubNet
- DPSNN: spiking neural network for low-latency streaming speech enhancement
- Phase-Aware Speech Enhancement with Deep Complex U-Net
- MP-SENet: A Speech Enhancement Model with Parallel Denoising of Magnitude and Phase Spectra
- Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks
- When Spiking Neural Networks Meet Temporal Attention Image Decoding and Adaptive Spiking Neuron
- Texture Synthesis Based Thyroid Nodule Detection From Medical Ultrasound Images: Interpreting and Suppressing the Adversarial Effect of In-place Manual Annotation
- Learning to Time-Decode in Spiking Neural Networks Through the Information Bottleneck
- Mutual Information Neural Estimation
- Learning Representations for Independence Testing
- Phase Transitions for the Information Bottleneck in Representation Learning