Continual Distillation of Ensemble Uncertainty into One Network

A deep ensemble, meaning several independently trained networks whose predictions are averaged, is the most reliable way we have to make a classifier know when it does not know. When its members disagree, the input is probably unfamiliar. The cost is that every prediction needs $M$ forward passes. A line of work removes this cost by distilling the ensemble into a single network that imitates both its average prediction and its disagreement. All of that work assumes the data arrive at once. In continual learning, where a model is trained on a sequence of tasks and must not forget earlier ones, the standard defense is functional regularization: the updating model is penalized for changing its outputs relative to a frozen snapshot of itself. That penalty protects the snapshot's predicted labels. Nobody protects its uncertainty.

We propose to treat the uncertainty signal as something that can be forgotten, and to guard it the same way continual learning guards class predictions. The deployed model is a single network with two outputs: class probabilities, and a scalar estimate of how much a small ensemble would disagree on the input. For each new task, a temporary ensemble is trained and its mean and disagreement are distilled into this network. Its snapshot then regularizes both outputs on old-task inputs, while its uncertainty is deliberately not used as a target on new-task inputs. The claim is that this retains near-ensemble out-of-distribution detection and calibration on old tasks at one forward pass. Methods that regularize only the class outputs lose both, even when they keep accuracy.

Core Claim

Calibration of Continual Learning Models shows that confidence degrades faster than accuracy across sequential tasks, even under soft-target distillation such as DER++. This is the motivating evidence. It says that matching old logits is not enough to preserve what the old model believed about its own reliability. Our hypothesis is stronger and testable: epistemic uncertainty, the kind that signals unfamiliar inputs, is forgotten even faster than calibration, because a mean-only regularizer never sees it. The remedy is to make the uncertainty an explicit output and regularize it functionally on the right inputs.

"The right inputs" is the subtle part. Functional Ensemble Distillation (FED) compresses an ensemble into a noise-conditioned generator by matching function distributions on mixup points. It found that ensemble members agree completely on their own training points, so disagreement must be taught on interpolated inputs. In continual learning this trap has a second side. The old snapshot, shown new-task images, reports high uncertainty, and correctly so, because those images are unfamiliar to it. Naively distilling that uncertainty would stop the model from ever becoming confident on the new task. Our design resolves both sides. Old-task uncertainty is preserved on buffer points and mixtures of buffer points, where the snapshot's disagreement is meaningful. New-task uncertainty is learned from a fresh ensemble on new data and mixtures of new data.

Method

At task $t$ the deployed student $f_\theta$ outputs logits $z_\theta(x)$ and an epistemic score $u_\theta(x)$. A fresh ensemble $E_t$ of $M$ members is trained on the task-$t$ data plus a small replay buffer $B$. Its targets are the mean probability $\bar p_E(x)$ and the mutual information $u_E(x) = H(\bar p_E(x)) - \frac{1}{M}\sum_m H(p_m(x))$. Here $H$ is entropy and $p_m$ is member $m$'s prediction; mutual information is the standard measure of member disagreement, separate from label noise. The frozen previous student $f_{\theta^*}$ supplies the old-task targets. The student minimizes

$$ \mathcal{L} = \mathcal{L}_{CE}(X_t) + \alpha\,\mathrm{KL}\big(\bar p_E \,\|\, p_\theta\big)_{\tilde X_t} + \beta\,\big(u_\theta - u_E\big)^2_{\tilde X_t} + \lambda\,\big\|z_\theta - z_{\theta^*}\big\|_{\infty,\,\tilde B} + \gamma\,\big|u_\theta - u_{\theta^*}\big|_{\tilde B} $$

In this loss, $\tilde X_t$ denotes task-$t$ inputs together with mixup interpolations among them. $\tilde B$ denotes buffer inputs and mixup interpolations among buffer items. The logit term is computed only over previously seen classes. The $L_\infty$ distance on logits follows Consistency is the Key to Further Mitigating Catastrophic Forgetting in Continual Learning, which found the maximum absolute logit difference to be the most effective consistency penalty on replay data. We reuse its technique.

The last term is the new ingredient: functional regularization of the uncertainty output. The asymmetry is the second ingredient. The fresh ensemble teaches uncertainty only on the new region. The snapshot preserves uncertainty only on the old region. Teachers are deterministic, because In Defense of the Learning Without Forgetting for Task Incremental Learning shows that stochastic teachers such as dropout create a moving target that increases forgetting. That paper also supplies our training recipe: residual backbones, augmentation, and no weight decay.

We use a scalar epistemic head rather than per-class intervals. Credal Ensemble Distillation for Uncertainty Quantification predicts $2C+1$ interval parameters, and the file notes that such interval heads scale poorly to many classes. A scalar head adds one output regardless of class count. The per-task ensemble is discarded after distillation. Training cost is roughly $M$ times higher per task, but deployment memory and latency stay at one network.

Relation to Prior Work

The static ensemble-compression methods supply the ambition, and they differ from ours in setting. FED is one. Improving Ensemble Distillation With Weight Averaging and Diversifying Perturbation (LatentBE) is another: it averages rank-one subnetworks inside one flat minimum into a single deterministic model. Contextual Similarity Distillation distills the variance of an infinite-width ensemble via kernel similarities. All three train once on fixed data, and none asks whether the distilled uncertainty survives later updates.

Calibrating Deep Ensemble through Functional Variational Inference (DE-GP) applies a functional prior to ensemble members but still runs all $M$ at test time. Liquid Ensemble Selection for Continual Learning manages an ensemble across tasks, but keeps every member and measures only accuracy. Continual learning with the neural tangent ensemble reads a single network as an internal ensemble and so pays no overhead. It constrains our expectations more than it supplies a method: its guarantees hold only in the lazy, wide-network regime, and it extracts no uncertainty. We work in the ordinary regime where features move.

Cascades reduce average ensemble cost rather than removing it: Window-Based Early-Exit Cascades and Towards Inference Efficient Deep Ensemble Learning. They are also tuned for a stationary test distribution. We treat them as out of scope for a fixed single-pass budget.

Research Program

The work has three parts. First, we measure "uncertainty forgetting." This is the per-task out-of-distribution AUROC (area under the ROC curve for separating unfamiliar from familiar inputs) and the per-task ECE (expected calibration error), tracked for each old task after every later task, for standard single-model continual learners. The measurement tests whether the phenomenon is real and larger than accuracy forgetting.

Second, we build the student above and ablate its two ingredients. Third, we test the method against the strongest single-pass alternatives and against the full-cost ensemble reference, under matched training compute.

Baselines

The naive reference is a single network trained with experience replay plus $L_\infty$ logit consistency, as in Consistency is the Key. Its uncertainty score is predictive entropy. That paper reports 20.47% final accuracy on Split-TinyImageNet with a 500-image buffer, against 10.08% for vanilla replay. We rerun it with our backbone, buffer, and epochs. Our student must stay within one accuracy point of it to show that the uncertainty terms do not cost plasticity.

The strongest single-pass calibration baseline is the same learner with Replayed Calibration from Calibration of Continual Learning Models. Replayed Calibration refits vector scaling on a stored validation buffer after each task. That paper reports ECE falling from 15.06% to 4.70% on EuroSAT with replay. We give it the same total memory as our replay buffer. Beating it means lower old-task ECE at equal accuracy, and higher old-task out-of-distribution AUROC, which post-hoc scaling cannot provide by construction.

The strongest overall competitor on detection is a distance-aware single model: SNGP from A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness, trained with the same replay and consistency loss. SNGP uses spectral normalization plus a Gaussian-process output layer, and reports 0.846 out-of-distribution AUROC on static CIFAR-100. That is the only single-model detection number for this setting in the file. There is no reported result on Lacuna for SNGP, or for any ensemble-distillation method, under continual learning. We establish those numbers ourselves with equal tuning budgets.

The cost-unconstrained reference is $M$ independent continual learners of the naive kind, ensembled at test time with $M$ times the latency. The goal is to recover most of its old-task AUROC at one pass. We will report the fraction recovered rather than claim to exceed it.

The ablations isolate the key ingredient. One removes the uncertainty-consistency term, which is mean-only continual distillation of the ensemble. One removes buffer mixup. One distills the snapshot's uncertainty on new-task inputs as well, the naive symmetric design.

Why not just…?

Why not just keep the small ensemble and pay $M\times$ at inference? If latency and memory are not constrained, this is the right answer, and we include it as the ceiling. The file's evidence is that the cost is real: FED reports over 5GB and 4.075 seconds for a large ensemble, against 43MB and 0.020 seconds for its generator. LatentBE reports 0.390 against 0.107 ms per image for four members. Unmanaged ensembles can also do worse under drift: roughly 59% against 62% for a single learner on Rotated MNIST in Liquid Ensemble Selection. The proposal is useful only if it closes most of the gap to this reference at one pass.

Why not just calibrate a single continual model post hoc? Temperature or vector scaling rescales confidence. It cannot create a disagreement signal for inputs the model has never seen, and Replayed Calibration needs a stored validation buffer. We expect it to match us on in-distribution ECE and lose on out-of-distribution AUROC. The Replayed Calibration baseline tests exactly this. If it also matches us on detection, the uncertainty head is unnecessary, and we would report that.

Why not just use SNGP with replay? This is the genuine threat. Distance awareness gives single-pass detection without any ensemble, and it adds no training cost per task. The risk on its side is that spectral normalization limits how far features can move to fit new tasks. The risk on ours is that its Gaussian-process layer already preserves distance awareness for old tasks. The SNGP baseline settles this directly. If it matches our old-task AUROC at equal accuracy, the ensemble-distillation step is not worth its training cost. We would then pivot to adding only the uncertainty-consistency term to SNGP's variance output.

Why not just give the single model the same training compute? Our method spends about $M$ times more training compute. We therefore also run the naive and SNGP baselines with $M$ times the epochs, and with a wider backbone at matched training FLOPs. This separates the benefit of the ensemble signal from the benefit of more compute.

Proposed Experiments

Benchmarks are class-incremental Split CIFAR-100 in 10 and 20 tasks; Split-TinyImageNet with a 500-image buffer, matching Consistency is the Key; and EuroSAT for calibration, matching Calibration of Continual Learning Models. Out-of-distribution sets for each old task are SVHN, plus classes from tasks not yet seen, which gives a natural unfamiliar set at every stage. We use ensembles of $M=5$ ResNet-18 members.

We report final accuracy, backward transfer, per-task ECE, NLL, and per-task out-of-distribution AUROC after every task. We summarize these as the area under the AUROC-forgetting curve. We also report measured latency and parameter count.

Three checks decide the question. First, mean-only distillation must show a steeper drop in old-task AUROC than in old-task accuracy. If it does not, uncertainty forgetting is not a problem worth solving. Second, adding the uncertainty-consistency term must flatten that drop without an accuracy loss beyond one point. Distilling snapshot uncertainty on new data must measurably hurt new-task accuracy, which would confirm the asymmetry. Third, at one forward pass, our student must beat SNGP-with-replay and Replayed Calibration on old-task AUROC under matched training compute, and recover a substantial fraction of the five-member continual ensemble's AUROC. Failing the third check against SNGP is the outcome we most need to rule out, and we will report it plainly if it occurs.