Uncertainty-Weighted Dark Replay: Distilling What the Old Model Knew, and How Sure It Was

Continual learning asks one neural network to learn a stream of tasks without forgetting earlier ones. The most reliable practical fix is experience replay, which keeps a small buffer of past examples and mixes them into each new training batch. The strongest simple variant also stores the model's output scores (logits) for each buffered example at the moment it was saved, and penalizes the current model for drifting away from them. This treats every stored score as equally trustworthy. That assumption is often false. When a sample is saved, the model may barely have trained on its class. It may also be emitting arbitrary scores for classes it has not yet seen. The replay penalty still pins the network to those scores as firmly as to its confident ones.

We propose to attach a cheap Bayesian posterior to the network's final linear layer, a Gaussian over the classifier weights fitted around the trained values (a last-layer Laplace approximation). This turns each stored logit vector into a mean plus a per-class variance. Replay then penalizes drift in proportion to how certain the old model was. The new model is held tightly where the old model had evidence and left free where it did not. The extra storage is one variance per class per buffered sample, a few hundred bytes next to a 3 KB image. The question is whether this per-sample, per-class epistemic signal adds anything beyond the obvious cheap fixes. Those fixes are softmax-confidence weighting, masking unseen-class logits, and normalizing the classifier. The note is built around answering that question decisively.

Core Claim

Dark Experience Replay (Dark Experience for General Continual Learning: a Strong, Simple Baseline) stores logits $\mu$ for each buffer sample and adds $\alpha\|z-\mu\|^2$ to the loss, where $z$ are the current logits. We replace this with a variance-normalized penalty:

$$\mathcal{L}_{\text{replay}} = \alpha \sum_{c=1}^{C} \frac{(z_c - \mu_c)^2}{\sigma_c^2 + \tau},$$

Here $C$ is the number of classes and $\sigma_c^2$ is the last-layer Laplace predictive variance of logit $c$, recorded when the sample entered the buffer. The term $\tau$ is a small floor, and the weights are renormalized to mean one per batch so that $\alpha$ keeps its usual scale. This is the diagonal of a Gaussian log-likelihood of the new logits under the old model's predictive distribution.

The claim is that this weighting improves both accuracy and calibration over DER++ at equal memory. It also predicts where the gain should come from: samples stored shortly after their class appeared, samples near class boundaries, and the logits of classes with little evidence at storage time. A shuffled-variance control should lose the gain.

Relation to Prior Work

The method itself is borrowed. Laplace Redux - Effortless Bayesian Deep Learning shows that a post-hoc Gaussian on the last layer gives useful uncertainty at near-zero cost. It supplies Kronecker-factored curvature and marginal-likelihood tuning of the prior precision. Its continual-learning experiments use the posterior only as a weight-space regularizer on Permuted-MNIST, with no buffer. With a Kronecker-factored last-layer posterior, the logit covariance of an input with feature vector $\phi$ is $(\phi^\top A^{-1}\phi)\,G^{-1}$. Here $A$ and $G$ are the two small factor matrices, so computing $\sigma_c^2$ at insertion time costs one feature-space quadratic form.

The closest prior work is Functional Regularisation for Continual Learning with Gaussian Processes (FRCL). FRCL treats the top layer as a Gaussian process and penalizes, by KL divergence, drift of the predictive distribution at optimized per-task inducing points. Our penalty is close to a diagonal version of that KL, and we say so plainly. We differ in four ways. Our memory is a single fixed-size reservoir buffer rather than inducing sets that grow with the number of tasks. Our distillation is interleaved with cross-entropy replay on real images. We need no task boundaries. And we target class-incremental CIFAR-scale streams, while FRCL reports only MNIST and Omniglot.

Several papers motivate the gap. Continual Learning in Deep Networks: an Analysis of the Last Layer finds that much forgetting lives in the classifier, through exploding norms and biases for recent classes. Error Sensitivity Modulation based Experience Replay: Mitigating Abrupt Representation Drift in Continual Learning (ESMER) and Learning Bayesian Sparse Networks with Full Experience Replay for Continual Learning (SNCL) find that backbone drift also matters. Logit-space targets with calibrated weights speak to both findings: they constrain the head's outputs without assuming a frozen backbone. SNCL uses Bayesian priors only for sparsity and leaves predictions deterministic. Deep Class-Conditional Gaussians for Continual Learning uses a generative Gaussian head and needs task identifiers.

Uncertainty Prioritized Experience Replay shows in reinforcement learning that epistemic signals beat raw error signals when data is noisy. We test the supervised analogue with a last-layer posterior instead of full ensembles. Two constraints shape the design. SOLA: Continual Learning with Second-Order Loss Approximation shows that inaccurate curvature makes long training harmful. Posterior Refinement Improves Sample Efficiency in Bayesian Neural Networks shows that Gaussian last-layer fits can be poor. Together they explain why we use the posterior only as a relative weighting with a variance floor, never as a hard constraint, and why we check its calibration directly.

Research Program

We would build the method on a standard reservoir-sampling replay setup with a ResNet-18 backbone. A running Kronecker-factored curvature estimate of the last layer is updated from current and buffered features at each step. Its cost is $O(d^2 + C^2)$ for feature dimension $d$ and class count $C$. The prior precision is re-tuned by marginal likelihood every fixed number of steps. When a sample enters the buffer, we store its image, label, logits, and the diagonal predictive variances $\sigma_c^2$.

The analysis has three parts. The first verifies that the stored variances behave as claimed. Logits of classes not yet seen should carry variance near the prior. Variance should fall as a class accumulates data. Samples near decision boundaries should have higher variance than prototypical ones. The second, addressing Gap 4 of the file, measures whether replay-flattened last-layer curvature makes the Laplace fit degenerate. We compare the Laplace predictive negative log-likelihood (NLL) against a refined posterior on held-out seen-class data at each task boundary. The third decomposes the gain by sample age, by seen versus unseen logit, and by boundary proximity.

Baselines

The naive reference points are plain experience replay and fine-tuning without replay. In plain replay, buffered images and labels are mixed into batches with cross-entropy only. ESMER reports plain ER at 21.91% on Seq-CIFAR100 with a 200-sample buffer and 22.20% on Seq-CIFAR10 with a 10-sample buffer.

The strongest direct competitor is DER++ from Dark Experience for General Continual Learning: a Strong, Simple Baseline. Our method changes only its logit-matching term. That paper reports 64.88% class-incremental accuracy on Seq-CIFAR10 with a 200-sample buffer and about 76% on MNIST-360. SNCL reports 61.93% for plain DER under the same buffer size and 66.16% for itself. ESMER reports 48.77% on Seq-CIFAR100 with a 200-sample buffer, which makes it the strongest replay method in the file on that benchmark. We also include ESMER and SNCL as stronger-system references, since they change more than the loss.

Fairness is enforced by bytes, not sample counts. The variances add $4C$ bytes per sample. DER++ and every weighted variant therefore receive extra buffer slots so that total memory matches. Every method gets the same backbone, epochs, augmentation, and an equal hyperparameter-search budget over $\alpha$, $\beta$, and learning rate on a validation split of the first tasks. Results use five seeds.

Three ablations isolate the key ingredient. The first, confidence-weighted DER++, weights each sample's logit penalty by its stored softmax confidence or negative entropy. The second, masked DER++, simply drops stored logits of classes unseen at storage time. The third, shuffled-variance, randomly permutes the stored variance vectors across buffer samples. It keeps the weight distribution intact but breaks the link between weights and samples.

Beating DER++ means a gain larger than two seed standard deviations in final average accuracy on Seq-CIFAR100. It also requires a lower expected calibration error (ECE) at equal memory, and the gain must survive the masked and confidence-weighted variants.

Why not just…?

Why not just weight DER's penalty by the old model's softmax confidence? This is the most serious threat. It costs one scalar and needs no curvature. Softmax confidence, however, mixes up "this sample is ambiguous" with "the model has not seen enough data". The reinforcement-learning evidence in Uncertainty Prioritized Experience Replay is that this conflation breaks prioritization under noise. Softmax confidence also gives one number per sample, not a separate certainty for each class. The confidence-weighted baseline is run head-to-head, and Noisy-Seq-CIFAR10 is the setting designed to separate the two signals. If confidence weighting matches us there and on Seq-CIFAR100, the Bayesian component is unnecessary and we would report that.

Why not just mask the logits of classes that did not exist when the sample was stored? This fix captures the most obvious source of bad targets, and it may account for a large share of any gain. The masked baseline and the per-logit decomposition settle this directly. Our claim survives only if variance weighting also helps on seen-class logits. That help should appear for recently introduced classes and boundary samples, and on MNIST-360, where there are no discrete class arrivals to mask by.

Why not just fix the classifier with cosine normalization or weight normalization, as in Continual Learning in Deep Networks: an Analysis of the Last Layer? Normalization removes the norm and bias imbalance, but it says nothing about which stored targets to trust, and it gives no uncertainty. The two ideas are complementary. We therefore run DER++ with a cosine classifier, with and without our weighting. If normalization alone closes the gap, the head imbalance was the whole story.

Why not use deep ensembles or MC dropout for the variances? Ensembles give better uncertainty, but they multiply training cost. Uncertainty Modeling for Machine Comprehension Systems using Efficient Bayesian Neural Networks and Posterior Refinement Improves Sample Efficiency in Bayesian Neural Networks both show that last-layer methods recover most of the uncertainty quality at a fraction of the cost. We include a dropout-head variance as a cheap alternative estimator to test whether the Laplace fit specifically matters.

Proposed Experiments

The benchmarks are Seq-CIFAR10 and Seq-CIFAR100 with buffers of 10, 200, and 500. We also run Seq-TinyImageNet, for which no reported result was found on Lacuna. MNIST-360 tests the boundary-free regime. Noisy-Seq-CIFAR10 with 50% symmetric label noise tests the epistemic-versus-aleatoric distinction; ESMER reports only a relative 116% gain over ER there, with no absolute number. Metrics are final average accuracy, average forgetting, and NLL and ECE on all seen classes. We also report wall-clock overhead per step against DER++.

There are three decisive checks. The first is that the full method beats both the confidence-weighted and the masked DER++ variants at equal bytes on Seq-CIFAR100 and on noisy Seq-CIFAR10. The second is that the shuffled-variance control falls back to DER++ level, which shows the gain comes from per-sample information rather than reweighting noise. The third is that the gain concentrates on young buffer samples and low-evidence logits, as the mechanism predicts.

If the first check fails while the stored variances still prove well calibrated, the outcome is a useful negative result. It would mean deterministic confidence already carries the information replay needs, and last-layer Bayesian inference in replay would be better spent elsewhere, for example on calibrated prediction or boundary-free task-change detection.