A spiking neural network is a network whose units talk to each other with brief binary pulses instead of continuous numbers. On neuromorphic chips its energy cost scales roughly with how many pulses it emits (Tavanaei et al., Jang et al.). Every bit of information such a network carries is therefore paid for in spikes, and that includes information it does not need. A spiking keyword spotter that also encodes who is speaking spends energy on speaker identity. It also generalizes poorly to new voices. The Heidelberg benchmark paper reports exactly this failure, with large accuracy drops on unseen speakers across all model families (The Heidelberg Spiking Data Sets).
We propose to train spiking speech classifiers with an explicit penalty on the mutual information (the amount of statistical dependence, linear or not) between the network's spike patterns and the speaker's identity, used together with an ordinary spike-count penalty. The hypothesis is that removing speaker information lets the network meet the same accuracy with fewer spikes, and also improves accuracy on unseen speakers. The mutual-information estimator exists only during training, so the deployed network is unchanged in size and cost.
Core Claim
Our claim is that, at a fixed spike budget, a spiking network trained to carry little speaker information reaches higher accuracy on unseen speakers than one trained with a spike-count penalty alone. It should also reach a given accuracy with fewer spikes. If that holds, mutual-information minimization does more than make a representation tidier. It becomes a way to reallocate a scarce energy budget toward the task.
The second, more technical claim concerns the estimator. Spike patterns are discrete, and speaker identity is a small discrete label. For that case, the standard mutual-information upper bound can be fit with a conditional model that is nearly free to train: a per-speaker table of firing rates over time bins. This removes most of the training-cost objection raised against mutual-information methods for lightweight networks.
Relation to Prior Work
Mutual-information minimization for disentanglement is well developed in conventional speech models, and these papers supply our method. CLUB introduced the contrastive log-ratio upper bound, which compares how likely a sample is under its own pairing versus random pairings. CLUB is the right tool for minimization, because lowering a lower bound such as MINE guarantees nothing about the true dependence. CLUB-style bounds have been used in speaker verification to strip device information (Disentangled Speaker Representation Learning via MI Minimization) and age information (Disentangling Age and Identity). SCD-Conformer showed that content leaks into speaker embeddings at both the frame and utterance level, using only linear correlation penalties. All of these work with dense floating-point embeddings and never consider energy. We reverse their direction: we remove speaker information from a content pathway, inside a spiking network, and measure the effect in spikes.
Two papers motivate the gap and also constrain the design. MI-TRQR is the closest energy-oriented use of mutual information in spiking networks. It measures redundancy between time steps and turns it into a non-differentiable dropout mask. It does not train weights against a bound and does not address nuisance factors. It also reports that information is concentrated in early time steps, and that estimating mutual information on spikes is costly during training. Both findings shape our choice of a cheap, label-conditioned estimator. The Hybrid SNN Guided VAE did separate label-relevant from nuisance factors in a spiking encoder, but it used adversarial training and lost accuracy when deployed on Loihi.
Decoupling Spatio-Temporal Coding used MINE only to measure information, and found that in spiking networks information shifts from activation magnitudes into spike timing. That is a warning that an estimator built on firing rates may miss dependence carried by timing. We treat this as a hypothesis to test rather than an assumption.
Research Program
The model is a spiking classifier for spoken digits or commands. It consists of a recurrent leaky integrate-and-fire network trained with surrogate gradients. A leaky integrate-and-fire neuron accumulates input, slowly leaks it, and fires when it crosses a threshold. As a second backbone we use the multiplication-free parallel spiking neuron of Multiplication-Free Parallelizable Spiking Neurons.
We take the spikes of the penultimate layer and bin them into $K$ time windows, giving a count vector $c \in \mathbb{N}^{K \times D}$ for $D$ neurons. We then train with:
$$ \mathcal{L} = \mathcal{L}_{\text{task}} + \alpha \, \hat{I}(c; s) + \beta \, \bar{n}_{\text{spikes}}, $$
Here $\mathcal{L}_{\text{task}}$ is cross-entropy on the word label, $s$ is the speaker label, $\bar{n}_{\text{spikes}}$ is the mean number of spikes per utterance across all layers, and $\alpha, \beta$ are weights.
The penalty $\hat{I}$ is the sampled CLUB bound with a spike-native conditional model. We model $q_\theta(c \mid s)$ as independent Poisson counts, one rate per speaker, neuron and time bin. This gives
$$ \hat{I}(c; s) = \frac{1}{N}\sum_{i=1}^{N}\Big[\log q_\theta(c_i \mid s_i) - \log q_\theta(c_i \mid s_{\pi(i)})\Big], $$
where $N$ is the mini-batch size and $\pi$ is a random permutation of the batch.
Fitting $q_\theta$ means updating a running table of mean counts per speaker. This is closed-form maximum likelihood, so there is no alternating optimization of a second network. Gradients reach the encoder through the counts via the same surrogate gradients used for the task loss.
The weakness of this estimator is known. CLUB is only an upper bound when $q_\theta$ fits well, and a per-bin Poisson model captures only speaker-dependent rate profiles, not correlations between neurons or fine timing. We therefore build a ladder of conditional models and compare them. The lowest rung is a Gaussian over utterance-level firing rates, which is the direct port of vCLUB. Next is the per-bin Poisson table. Above that is a small Bernoulli model over fine time bins, conditioned on a learned speaker embedding, which can capture timing. Each rung must be checked against independent measurements of leakage, described below, because a loose estimator can report low dependence while leakage persists.
Leakage is measured three ways after training, with the network frozen. We train a speaker probe (a recurrent classifier on the spike trains) and report its accuracy. We report a MINE estimate of $I(c; s)$, used only as an evaluation lower bound. We also report per-neuron speaker selectivity against firing rate, which directly tests whether spikes were being spent on speaker identity.
Energy is reported as spike counts and accumulate operations per inference. A synaptic event in a spiking network costs an addition rather than a multiply-accumulate (Wu et al.), so we convert operation counts to joules using the same per-operation constants as the cited spiking papers. This is a proxy, not an on-chip measurement, and we will say so in the results.
Baselines
The naive reference point is the same backbone trained with task loss only ($\alpha = \beta = 0$). It tells us how much speaker information an unconstrained spiking network carries, and what that information costs in spikes. For orientation, the Heidelberg paper reports a recurrent spiking network at 71.4% on the Spiking Heidelberg Digits dataset and 50.9% on Spiking Speech Commands, and 83.2% on the digits after augmentation and size tuning (Heidelberg).
The strongest existing accuracy result in the file is the multiplication-free parallel spiking neuron at 95.50% on the digits (Multiplication-Free PSN). We will reproduce it, add our penalty on top, and run both versions with the same data, number of epochs and hyperparameter search budget. We found no reported result on Lacuna for an unseen-speaker split or for energy per inference on the digits, so the baseline numbers on our split will be our own reproductions.
The baseline that actually decides the question is the spike-count penalty alone ($\alpha = 0$, $\beta$ swept). Comparing against it separates "fewer spikes" from "fewer spikes spent on speakers". Beating it means the full accuracy-versus-spike curve with $\alpha > 0$ lies strictly above the $\alpha = 0$ curve on unseen speakers, across seeds.
The adversarial competitor is a speaker classifier attached through gradient reversal, given the same tuning budget. This is the method family used by the Guided VAE and the one CLUB outperformed in a conventional-network domain-adaptation setting (90.6% versus 71.1% for DANN on SVHN to MNIST, CLUB). The linear competitor is a cross-correlation penalty between spike counts and one-hot speaker labels, in the spirit of SCD-Conformer.
Two ablations isolate the key ingredient. The first is the estimator ladder, which asks whether a spike-native conditional model matters at all compared with Gaussian vCLUB on firing rates. The second is a shuffled-speaker control, where the penalty uses permuted speaker labels. This control rules out the possibility that any gain is just extra regularization.
Why not just…?
Why not just penalize spike counts or cut time steps? This reduces energy, but nothing in it targets what the removed spikes encoded. The regularize-and-cutoff work cuts time steps by 1.64 to 1.95× with near-zero accuracy loss (Optimizing event-driven SNN), but only on matched-distribution tests. The Heidelberg results also show that collapsing timing into counts is costly (about 55% for a count-only linear classifier on the digits). Our main comparison is this baseline swept densely. If its curve matches ours on unseen speakers, the proposal fails, and we will report that.
Why not just use an adversarial speaker classifier? This is the most serious threat, since with a discrete nuisance label the two methods are close relatives. The evidence against adversarial training comes from conventional networks (the CLUB versus DANN comparison above) and from the Guided VAE's fragility, not from spiking speech models. Our head-to-head comparison at matched tuning budget, with probe accuracy and seed variance reported, will settle it. If gradient reversal matches CLUB, the energy finding still stands, and we will present the estimator choice as a secondary result.
Why not just apply vCLUB with a Gaussian model on firing rates? This may work as well as our spike-native estimator, because speaker identity in cochlear spike codes may live mostly in rates. It is the lowest rung of our estimator ladder. If it matches the binned models on probe accuracy and on unseen-speaker accuracy, we will recommend it and drop the claim that a discrete estimator is needed. The timing-shift finding in Decoupling Spatio-Temporal Coding gives a reason to expect that it misses timing-borne leakage, which the fine-bin probe would reveal.
Why not just add data augmentation or more speakers? Augmentation helped the Heidelberg recurrent spiking network, but it changes the training distribution and does not reduce spike cost. We will apply the same augmentation to every arm, so any gain from the penalty is measured on top of it.
Proposed Experiments
We use the Spiking Heidelberg Digits (12 speakers) and Spiking Speech Commands, which both come with speaker metadata. We report the official test split, and we also report leave-speakers-out splits where whole speakers are held out from training, rotated across folds. The primary result is a set of accuracy-versus-spikes curves on held-out speakers, one per arm, swept over $\beta$ and over $\alpha$, with five seeds each.
There are three decisive checks. First, at matched mean spike count, the mutual-information arm must show higher unseen-speaker accuracy and lower speaker-probe accuracy than the spike-penalty-only arm. Second, the spikes saved must be concentrated in neurons that were speaker-selective in the naive model. This is the mechanism claim; if savings are spread evenly, the "paying for identity" story is wrong even if accuracy improves. Third, accuracy on seen speakers should not fall, given that MI-TRQR found information concentrated early and the Heidelberg paper found full-sequence timing necessary. A drop would indicate over-pruning, and we would raise the bin resolution or lower $\alpha$.
A secondary experiment repeats the setup on DVS128 Gesture, an event-camera dataset with 29 subjects and 4 lighting conditions. The nuisance label there is subject identity, against the 87% Guided-VAE reference (Hybrid SNN Guided VAE). This tests whether the effect holds beyond audio.
References
- Deep learning in spiking neural networks
- Spiking Neural Networks - Part I: Detecting Spatial Patterns
- The Heidelberg Spiking Data Sets for the Systematic Evaluation of Spiking Neural Networks
- CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information
- Mutual Information Neural Estimation
- Disentangled Speaker Representation Learning via Mutual Information Minimization
- Disentangling Age and Identity with a Mutual Information Minimization for Cross-Age Speaker Verification
- SCD-Conformer: Semantic Content Disentanglement for Text-Independent Speaker Verification
- MI-TRQR: Mutual Information-Based Temporal Redundancy Quantification and Reduction for Energy-Efficient Spiking Neural Networks
- Encoding Event-Based Data With a Hybrid SNN Guided Variational Auto-encoder in Neuromorphic Hardware
- Decoupling the Contributions of Spatio-Temporal Coding: From ANNs to SNNs
- Multiplication-Free Parallelizable Spiking Neurons with Efficient Spatio-Temporal Dynamics
- Deep Spiking Neural Networks for Large Vocabulary Automatic Speech Recognition
- Optimizing event-driven spiking neural network with regularization and cutoff