A deployed classifier usually has no labels at test time, so nobody knows when its accuracy starts to slip. Methods exist that guess accuracy from unlabeled inputs, but they assume a fixed batch of test data. Separately, a classic result from computational neuroscience shows that a single leaky spiking neuron can act as an optimal detector of abrupt change. We propose to combine these two lines into one streaming estimator. Each incoming example yields a cheap per-example guess of whether the model is correct. That guess feeds a small bank of leaky accumulators, each forgetting the past at a different rate, so the bank holds running accuracy estimates over many time horizons at once. A change-detection unit, modeled on the spiking change detector, then decides which horizon to trust at each moment. The result is a per-example, constant-memory estimate of current accuracy. It tells an operator how much accuracy has dropped, not merely that the inputs look different.
This matters because the two tools now available answer different questions. Drift detectors such as the incremental Kolmogorov-Smirnov test raise alarms on changes in the input distribution even when the model is still correct. Accuracy estimators such as Average Thresholded Confidence and the Projection Norm need a static target set, and in ProjNorm's case at least a thousand samples plus fine-tuning. The neuroscience thread adds two things. It supplies the mechanism: leaky integration and threshold crossing. It also supplies a hard constraint: only local, leaky, threshold operations are allowed, which keeps the estimator cheap enough to run on every example. Along the way the proposal addresses an open problem the original spiking change-detection paper flagged: how such a detector can work when the post-change statistics are not known in advance.
Core Claim
Estimating accuracy under drift is a filtering problem over a hidden, time-varying accuracy, not a batch statistic. The core claim has three parts. First, take the per-example correctness proxy used by Average Thresholded Confidence, $c_t = \mathbb{1}[s(x_t) > \tau]$, where $s$ is the model's confidence on input $x_t$ and $\tau$ is the threshold calibrated on labeled source data. Second, feed it into $K$ leaky integrators with log-spaced forgetting rates $\lambda_k$:
$$m_k(t) = (1-\lambda_k)\, m_k(t-1) + \lambda_k\, c_t, \qquad k = 1,\dots,K.$$
This bank is the discrete-time analogue of the decay-rate spectrum that Computational Model for Episodic Timeline Based on a Spectrum of Synaptic Decay Rates uses to encode a log-compressed record of the past. Third, read the bank out with a weighted average $\hat{a}(t) = \sum_k w_k(t)\, m_k(t)$. Each weight $w_k(t)$ is proportional to a leaky running Bernoulli likelihood of recent indicators under estimate $m_k$. The result is a soft posterior over "how long has the current regime lasted."
When the stream has been stable, the slow integrators win and the estimate has low variance. Right after an abrupt drop, the fast integrators explain recent indicators better and take over. The fast integrators also serve as the online estimate of the unknown post-change statistics. That is exactly the quantity Optimal Change-Detection and Spiking Neurons had to assume was known, since its update $\Phi_t = a(\Phi_{t-1}+q)$ requires the before-and-after densities and the hazard rate $q$ as fixed inputs. The prediction that would make this worth doing is specific. On streams that mix gradual and abrupt drift, the bank should beat every single forgetting rate, including one tuned in hindsight, because the best rate differs from regime to regime.
Positioning Against Prior Work
The accuracy-estimation side supplies the per-example signal. Leveraging unlabeled data to predict out-of-distribution performance introduces Average Thresholded Confidence and proves that accuracy estimation is impossible without assumptions on the shift. We inherit both its indicator and its limits: no unlabeled method can see a change in the label rule when confidence stays the same. Predicting Out-of-Distribution Error with the Projection Norm fine-tunes a model copy on pseudo-labels and is more robust, including under adversarial attack. It needs batches of about a thousand samples, so we use it as an expensive periodic reference, not a competitor at per-example latency. On the Importance of Gradients for Detecting Distributional Shifts in the Wild and Estimating Accuracy from Unlabeled Data: A Bayesian Approach provide alternative per-example proxies: a gradient norm, and agreement between classifiers. Both can be swapped in as $c_t$. All four papers estimate over a fixed target set. We change the temporal estimator on top of them, not the proxy.
The drift side motivates the problem and constrains the design. Fast Unsupervised Online Drift Detection Using Incremental Kolmogorov-Smirnov Test makes input-distribution tests cheap, but it is model-blind and univariate, so it flags harmless shifts. Pitfalls of Unlabeled Disagreement-Based Drift Detection in Streaming Tree Ensembles is the closest existing bridge between drift and unlabeled performance signals. It turns disagreement into a binary drift alarm and finds that the alarm works for multilayer perceptrons but lags for incremental trees. We differ in producing a continuous accuracy estimate rather than an alarm. We also take its negative result as a warning: the proxy's responsiveness depends on the architecture. Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time documents the stakes. Yearbook accuracy falls from about 98% to about 79% on future years, and MIMIC-IV mortality AUC falls from 0.90 to 0.72. Its finding that plain empirical risk minimization is hard to beat means the models we monitor will be ordinary ones.
The neural side supplies the mechanism and the locality constraint. The decay-spectrum memory paper and the leaky-neuron change detector are reused directly. Neural Implementation of Bayesian Inference in Population Codes explicitly leaves non-stationary stimuli open. Our bank is one concrete answer for the scalar case. We do not claim the brain does this. We claim that the operations the estimator needs (leak, accumulate, threshold, locally gated readout) are ones known to be implementable in spiking units, and that this constraint yields an estimator cheap enough to run on every example.
Research Program
The work proceeds in three stages. The first stage builds the estimator for a generic per-example proxy, with $K$ between 8 and 16 and rates spaced geometrically from about $10^{-1}$ to $10^{-4}$. It derives the readout weights from a leaky likelihood comparison. It also includes a sparse-label channel: when the change unit's posterior on a recent change crosses a threshold, a small number of labels are requested to recalibrate $\tau$. This answers the hidden label-rule change that no unlabeled proxy can detect, and it mirrors how the IKS paper pairs alarms with label queries. The second stage analyzes the bias and variance of the bank under step and ramp drift in accuracy. The aim is to give the condition under which mixing timescales strictly beats the best single rate, which should be the case when step changes are rare relative to the slowest useful horizon. The third stage, secondary, implements the bank and gate as leaky integrate-and-fire units in a small simulation. Membranes act as integrators, a threshold unit computes $c_t$, and a Yu-style unit computes the gate. This stage checks that the spiking version reproduces the floating-point estimate within a stated tolerance.
Baselines
The naive reference is to report source validation accuracy forever. Any estimator that cannot beat it on Wild-Time's future years is useless. The second reference is Average Thresholded Confidence applied cumulatively to everything seen since deployment, which is the literal batch method run online.
The strongest practical competitor, and the key ablation, is Average Thresholded Confidence with a single exponential moving average whose rate is tuned in hindsight on held-out streams from the same benchmark. This is exactly one integrator of our bank. Beating it isolates the contribution of the timescale spectrum plus the gating. A fixed-length sliding window with tuned length is run alongside it.
The third competitor is a detect-and-reset pipeline. It runs the incremental Kolmogorov-Smirnov test on the stream of confidence scores, which is univariate and so within that method's scope, and restarts the accuracy estimate whenever an alarm fires.
As a quality reference, ProjNorm is run on windows of 1,000 examples. On CIFAR-100 it reports $R^2 = 0.972$ against 0.939 for Average Thresholded Confidence (ProjNorm paper), but in a static setting. We found no reported result on Lacuna for streaming per-example accuracy estimation on Wild-Time, the IKS streams, or temporally scheduled CIFAR-10-C.
All methods share the same deployed models, the same proxy, the same source calibration set, and an equal tuning budget of grid search over each method's free parameters on held-out streams. Label queries are counted identically. Beating a baseline means lower time-averaged absolute error between estimated and true running accuracy, with 95% intervals that do not overlap across seeds and stream orderings, at equal or lower memory per step.
Why not just…?
Why not just use a well-tuned moving average of the confidence indicator? This is the real threat, since a single integrator is a special case of our method. Under purely gradual drift, a tuned moving average may match the bank, and we expect it will. The bank should win only when regimes mix: a fast rate is too noisy during stable periods, and a slow rate lags after abrupt drops. The mixed-regime experiments below settle the question. If the hindsight-tuned single rate comes within noise of the bank on held-out streams, we will report that the spectrum adds nothing.
Why not just detect drift with the incremental Kolmogorov-Smirnov test and reset the estimate? A hard reset discards good history whenever the input distribution shifts harmlessly. The IKS paper's own design makes it model-blind, so such false resets are expected. It also throws away the gradual-drift signal entirely between alarms. Our gate is a soft version of reset that reads timescale evidence from the correctness proxy itself. The head-to-head comparison measures how often each method resets or re-weights on shifts that leave true accuracy unchanged.
Why not just run ProjNorm periodically on buffered windows? It is more robust than confidence proxies, including under attack where logit methods predict errors below zero (ProjNorm paper). However, it needs about a thousand samples plus a fine-tuning run per estimate, which forces a lag of at least one window. We include it as a reference and report the latency-accuracy curve. If it dominates at all latencies that matter for a given stream, the bank's value is limited to low-compute or per-example settings, and we will say so.
Why not just buy labels? The IKS paper reaches parity with full labeling using 42% to 47% of labels, which is still expensive. We report error as a function of label budget, including zero labels.
Proposed Experiments
We run three stream families. The first is Wild-Time under its Eval-Stream protocol, using Yearbook, FMoW-Time, HuffPost, arXiv, and MIMIC-IV, with the true running accuracy computed from held-back labels. The second is the six IKS streams: Arabic, Posture, Bike, Keystroke, Insects, and Abrupt Insects. The third is controlled CIFAR-10-C schedules that interleave ramps of corruption severity with abrupt corruption switches and stable plateaus. Only this family gives exact ground-truth accuracy at every step. On some schedules we add label flips to simulate a hidden change in the label rule.
Metrics are time-averaged absolute error of the accuracy estimate. For drops of a stated magnitude in true accuracy, we also report detection delay and false alarms. We report memory and compute per step, and labels used.
The decisive check is the mixed-regime CIFAR-10-C and Wild-Time comparison against the hindsight-tuned single moving average. The idea survives only if the bank wins there without losing on stable or purely gradual streams. A second check targets label flips. We expect every unlabeled method, ours included, to miss them. The sparse-label channel should recover within a stated number of queries, which tests whether the gate spends labels where they matter. A third check swaps the proxy among confidence threshold, gradient norm, and two-model disagreement. The aim is to see whether the bank's gains are independent of the proxy, and whether disagreement inherits the lag that the tree-ensemble pitfalls paper found for rigid models. The spiking simulation is judged only on whether it matches the floating-point estimator. Its role is to show that the method respects the locality constraint, not to compete on accuracy.
References
- Computational Model for Episodic Timeline Based on a Spectrum of Synaptic Decay Rates
- Optimal Change-Detection and Spiking Neurons
- Leveraging unlabeled data to predict out-of-distribution performance
- Predicting Out-of-Distribution Error with the Projection Norm
- On the Importance of Gradients for Detecting Distributional Shifts in the Wild
- Estimating Accuracy from Unlabeled Data: A Bayesian Approach
- Fast Unsupervised Online Drift Detection Using Incremental Kolmogorov-Smirnov Test
- Pitfalls of Unlabeled Disagreement-Based Drift Detection in Streaming Tree Ensembles
- Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time
- Neural Implementation of Bayesian Inference in Population Codes