Energy per Reliable Attention Switch: Calibrated Stopping for Spiking EEG Attention Decoders

A hearing aid that follows a listener's attention has to decide which of two talkers the wearer is listening to, using brain signals recorded by electroencephalography (EEG, electrodes on the scalp or in the ear). Two recent lines of work each report progress on part of this problem. Accuracy on very short slices of EEG has become striking, with up to 98.4% reported on 0.1-second windows. Separately, spiking neural networks, which communicate with sparse binary events instead of dense floating-point numbers, report several-fold lower estimated energy per decision. A device, however, does not need a correct label for each tenth of a second. It needs to switch the amplified talker soon after the listener switches, without flipping back and forth by mistake, and to do so on a milliwatt battery. None of the published numbers measures that. They also come without uncertainty estimates that respect how EEG data is organized: a few dozen subjects, a few long trials each, and thousands of strongly correlated windows cut from those trials.

This note proposes one quantity and one mechanism, and argues that each depends on the other. The quantity is the energy a decoder spends per correctly detected attention switch, measured at a fixed rate of false switches. Its confidence interval comes from a resampling procedure that draws whole subjects and whole trials, never individual windows. The mechanism is an evidence-accumulating spiking decoder that stops as soon as it is confident enough. Its stopping threshold is set by resampling a subject's short calibration recording, so that the false-switch rate on new data comes out at the target. The hypothesis is that adaptive stopping, not per-window accuracy, is where event-driven hardware earns its energy advantage. The note also claims that answering this cleanly requires the resampling machinery the statistics literature built for skewed, product-shaped estimators.

What the three directions contribute, and where the gap is

Low-latency attention decoding has moved away from reconstructing the speech envelope. That approach needs 20 to 60 seconds of data per reliable decision (An Auditory Paradigm for Brain-Computer Interfaces; EEG-assisted modulation of sound sources in the auditory scene). The field now classifies the spatial side of attention, left versus right, which is visible in scalp topography almost immediately. Within this trend, PCRNet adds phase-aware complex spectral processing and reports 98.4% on KUL at 0.1 s with 30,000 parameters and 95.05 MFLOPs per 1-second window. ListenNet reaches 0.01M parameters and adds covariance alignment across subjects. These results motivate the proposal, and they also raise a warning. Geirnaert et al. showed that a spatial convolutional decoder that won on one dataset was "not competitive" on the reverberant DTU dataset. The same review also argues that the right figure of merit is the minimal expected switching duration (MESD, the shortest time within which a decoder can reliably follow a switch), not per-window accuracy. Very high per-window accuracies on short slices therefore need to be tested for whether they carry over to held-out trials and whether they translate into fast, stable switching.

On the spiking side, BSAnet is an end-to-end spiking attention decoder with about 900 parameters. It reports 93.7% on KUL at 1 s and 84.7% at 0.25 s. S²M-Former reports 85.28% on DTU at 2 s, 0.0779 mJ per inference (5.8× below the DBPNet decoder), and a surprising result: it beats its own non-spiking twin, SM-Former. Both energy figures are theoretical, obtained as synaptic operation counts multiplied by a per-operation cost, and both are reported per fixed window. That per-window framing hides the main structural property of spiking hardware. Dynamic energy scales with activity, so work can stop the moment a decision is made (Tavanaei et al.). A dense convolutional network processing fixed windows gains nothing from stopping early inside a window. An event-driven accumulator does. These papers supply the decoders we reuse. The per-window energy accounting they rely on is the limitation we address.

The resampling direction supplies the statistical method. The product of two regression coefficients in mediation analysis has a skewed, heavy-tailed sampling distribution. Normal-theory tests such as Sobel's therefore miscalibrate, and the standard fix is the bias-corrected percentile bootstrap, used routinely in work such as Chain relationship between cumulative ecological risk and physical activity pattern on college students mediated by exercise atmosphere and self-control. Energy per switch has the same shape of problem. It is a product of a detection time, which is itself a nonlinear function of accuracy with a long right tail when accuracy nears chance, and an activity-dependent energy rate. We take the bias-corrected percentile interval from that literature. We change the resampling unit from individual observations to subjects and trials, because consecutive EEG windows are not exchangeable.

The closest resampling work in EEG is Informed Bootstrap Augmentation. It weights trials by an event-related-potential quality score when resampling synthetic training averages, raising accuracy from 68.34% to 71.25% on PRECOG. That paper uses resampling to create training data for event-locked responses. We use resampling for inference and for calibrating a decision threshold on continuous, non-event-locked data, where its quality metric has no counterpart.

Core Claim

A spiking attention decoder should be judged, and designed, by the energy it spends per correctly followed attention switch at a fixed false-switch rate. Under that criterion, the useful ingredient is a stopping rule calibrated per subject by subject-and-trial resampling. We expect such a rule to reduce energy per switch relative to fixed-window decoding by a margin larger than the spiking-versus-dense difference reported so far. We also expect it to change the ranking between decoders that look similar on per-window accuracy. Both claims can fail, and the experiments below are designed to show which.

Research Program

The decoder is an existing spiking network, S²M-Former from its public code (JackieWang9811/S2M-Former) and BSAnet as a much smaller second case. It runs on short consecutive windows of length $\Delta$ (0.1 to 0.25 s). Each window produces a probability $p_k$ that attention is on the left. A streaming accumulator keeps a running log-evidence score:

$$L_k = \max\!\Big(-\eta,\ \min\big(\eta,\ \lambda L_{k-1} + \log\tfrac{p_k}{1-p_k}\big)\Big).$$

Here $\lambda \in (0,1]$ is a forgetting factor, which the leaky membrane of a spiking output neuron implements directly. The device switches the amplified talker when $L_k$ reaches $+\eta$ or $-\eta$ and the side differs from the current one. After a switch, upstream layers can be gated off until evidence against the current side begins to build. This is where event-driven hardware can save energy that a dense decoder cannot.

The threshold $\eta$ is the quantity to calibrate. With only a few minutes of calibration EEG from a new user, we resample whole calibration trials with replacement. On each resample we splice synthetic switch sequences from segments with opposite attended sides, the construction Geirnaert et al. use for switching analysis. We then choose the smallest $\eta$ whose bias-corrected upper confidence bound on the false-switch rate stays below a target $\alpha$, for example one false switch per five minutes. Resampling trials, not windows, is the essential design choice. Window-level resampling treats hundreds of correlated windows as independent, which would make the bound far too optimistic.

The evaluation quantity is

$$E_{\text{switch}} = \mathbb{E}[T_{\text{detect}}]\cdot \bar{r}_{\text{ops}} \cdot e_{\text{op}},$$

where $T_{\text{detect}}$ is the time from a true switch to the device's switch, $\bar{r}_{\text{ops}}$ is the average synaptic operations per second during that interval (measured spike activity for spiking models, dense MAC count for non-spiking models), and $e_{\text{op}}$ is the per-operation energy used by S²M-Former, with separate values for accumulate-only and multiply-accumulate operations (Wu et al.). We report it only at a matched false-switch rate. Its confidence interval comes from a two-stage bootstrap: resample subjects, then trials within each subject, then compute a bias-corrected percentile interval over 5,000 replicates, the replicate count used in the mediation literature (Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control). Because every quantity is a synaptic-operation estimate, we treat the cross-model energy comparisons as relative, not absolute. The chip-level deployment gap flagged for both spiking decoders is not closed here. We state this as a limitation and do not claim to have solved it.

Baselines

The naive reference is the same decoder deciding on independent fixed windows, 1 s and 2 s, with each window's argmax driving the device directly and no accumulation. Its energy per switch is the per-window energy multiplied by the number of windows until a correct stable decision. This is how the existing per-inference numbers would translate into device behavior, so beating it shows that the accumulator adds something.

The strongest spiking competitor is S²M-Former in its published configuration. The file reports 85.28% accuracy on DTU with 2-second windows and 0.0779 mJ per inference (S²M-Former). We retrain it from the public code on our trial-held-out splits, with the tuning budget its authors describe, and give it the same accumulator and calibration as our system. Any gain we report therefore comes from stopping and calibration, not from a weaker reproduction.

The strongest non-spiking competitor is PCRNet: 98.4% on KUL at 0.1 s, 95.05 MFLOPs per 1-second window (PCRNet). For DTU, the file has no isolated percentage, so no reported result was found on Lacuna for PCRNet on DTU. ListenNet is the smallest dense model at 0.01M parameters; no absolute accuracy for it is reported on Lacuna, only margins over prior work. Both are retrained on identical splits with matched hyperparameter search, and both get the same accumulator. Energy for them is counted as dense multiply-accumulates, since they cannot skip work within a window.

Beating these baselines means that the spiking decoder with calibrated stopping has an $E_{\text{switch}}$ whose 95% bootstrap interval lies entirely below that of both the fixed-window spiking decoder and the best dense decoder with the same accumulator, at a matched false-switch rate, on held-out subjects.

Three ablations isolate the key ingredients. The first keeps the accumulator but sets $\eta$ from a Gaussian approximation to the calibration false-switch rate, the analogue of the Sobel test, instead of the trial bootstrap. The second calibrates with window-level resampling. The third runs the accumulator on SM-Former, the non-spiking twin, to separate the effect of spiking dynamics from the effect of adaptive stopping.

Why not just…?

Why not just report mean accuracy and a standard deviation across subjects, as current papers do? Accuracy on pre-cut windows does not tell a hearing aid how long a switch takes or how often it flips wrongly. Geirnaert et al. make this argument for MESD, and our energy quantity extends it. A mean and standard deviation also treat a skewed functional as symmetric, the same failure the mediation literature documents for the Sobel test. The Gaussian-threshold ablation directly measures how far normal theory misses the target false-switch rate on held-out subjects.

Why not just smooth fixed-window outputs with a majority vote or a two-state hidden Markov model? This is a real threat. Much of any gain might come from temporal smoothing, independent of spiking or resampling. We therefore run a hidden-Markov smoother, with its switch prior tuned on the same calibration data, as an additional arm on every decoder. If it matches calibrated stopping on both false-switch coverage and energy per switch, the contribution reduces to the evaluation protocol, and we will report it that way. We expect it to fall short on energy because it still processes every window at full cost, and possibly on coverage because its tuned prior has no guarantee on held-out data. Only the experiment can settle this.

Why not just apply informed bootstrap augmentation to boost the spiking decoder's accuracy? Its trial weights come from event-related-potential differences between conditions (Informed Bootstrap Augmentation). Continuous two-talker listening has no stimulus-locked peaks to score. Higher per-window accuracy would also shorten detection time without addressing false-switch control. It is complementary at best, and we leave it aside.

Why not just deploy PCRNet, given its 98.4% at 0.1 s? If that accuracy survives trial-held-out splits, PCRNet with the accumulator may reach the lowest detection time of any model, and dense computation may still cost less per switch than a spiking model that needs longer to reach threshold. Our comparison is set up to allow this outcome. If it occurs, it is a useful negative result for the case for neuromorphic hardware in this task.

Proposed Experiments

We use KUL and DTU, the latter with its anechoic and reverberant conditions analyzed separately, plus AV-GC-AAD as a third site. All splits hold out entire trials, and a leave-subjects-out split is used for calibration tests. Switch sequences are spliced from held-out segments with opposite attended sides, as in MESD analyses. The primary outcomes are empirical false-switch rate relative to target, mean detection time, and $E_{\text{switch}}$, each with a two-stage bootstrap interval.

Three checks decide the question. First, on held-out subjects, the trial-bootstrap threshold keeps false switches at or below $\alpha$ while the Gaussian and window-resampling thresholds exceed it. If all three coverages match, the resampling ingredient is unnecessary. Second, we re-measure the per-window accuracy gap between S²M-Former and SM-Former, and PCRNet's short-window accuracy, under trial-held-out splits. If these numbers drop sharply, the literature's per-window rankings need revision, and energy-per-switch rankings are the ones to trust. Third, the headline comparison: whether calibrated stopping on a spiking decoder gives an energy-per-switch interval clearly below both its fixed-window version and the best dense decoder with the same accumulator, and whether that advantage holds in DTU's reverberant rooms, where spatial decoders have failed before.