A model that learns classes a few at a time and must later recognize any of them without being told which batch an image came from faces a routing problem. It has to work out which earlier "task" (group of classes learned together) the image belongs to before it can classify it. Theory shows this routing step is essentially out-of-distribution detection, meaning the job of deciding whether an input comes from the distribution a model was trained on. Good routing combined with good within-task classification is necessary and sufficient for solving the whole problem. Reconstruction-based anomaly detection offers a natural router. Train a generative model on each task, then send the image to whichever task's model can best "heal" it back into something familiar. The only attempt at this in continual learning failed on natural images. Its per-task autoencoders reconstructed foreign images almost as well as native ones, a failure the anomaly-detection literature calls the identical shortcut.
We propose a fix that is specific to the continual setting. One frozen denoising network, a diffusion model's core component that removes noise from a corrupted image, is shared across all tasks, and each task adds a small isolated adapter to it. To score a test image against task $k$, we corrupt it once, with a fixed mask and a fixed noise draw. We then compare how much better task $k$'s adapter restores it than the shared base network does on that same corruption. Subtracting the base network's error cancels what every model finds easy, such as smooth or simple images. Using identical corruptions across tasks puts every task's score on the same scale without a calibration buffer. The claim is that this paired, base-relative residual turns reconstruction from a router that collapses on CIFAR-scale images into one that is competitive with feature-distance routing, without storing any past exemplars.
Core Claim and Positioning
The motivating theory comes from A Theoretical Study on Solving Continual Learning. It proves that class-incremental learning (CIL) reduces to within-task prediction plus task-level OOD detection. It then validates this by stitching parameter-isolation classifiers to discriminative OOD detectors (ODIN, CSI). It never tests a reconstruction-based detector. It also needs a small memory buffer (+c) to align scores across independently trained task heads, and that buffer is the problem our pairing targets.
The direct predecessor is SurpriseNet. It trains one autoencoder per task inside a capacity-partitioned network and routes each test image by lowest reconstruction error. On raw pixels it reaches only 16.49% on Split-CIFAR100. Swapping in a frozen pretrained ResNet18 and reconstructing features instead (SurpriseNetE) lifts this to 48.15%. That result motivates us: reconstruction routing is fragile. It also constrains us: moving to features helps, so any pixel-level method must justify itself against feature-space routing. We differ from SurpriseNet in three ways. Each task's scorer is an adapter on a shared denoiser rather than an independent autoencoder. The corruption, a checkerboard mask plus noise, blocks copying. And the score is a paired difference against the shared base rather than a raw error.
The method borrows two pieces from the anomaly-detection side. The masking-and-inpainting probe comes from LMD. LMD lifts an image off the data manifold with a checkerboard mask, inpaints it with a diffusion model trained on in-distribution data, and scores it by perceptual distance. It reports 0.985 ROC-AUC on CIFAR-100 versus SVHN. The idea of scoring at a few noise levels with single network evaluations, instead of full iterative sampling, comes from Multiscale Score Matching, which classifies vectors of score norms across noise scales.
Both papers address far-OOD pairs, where the two datasets look very different. Task-versus-task routing within CIFAR-100 is near-OOD: classes share colour statistics, textures and backgrounds. The identical-shortcut analyses in DRÆM and DeScarGAN warn that unconstrained reconstructors reproduce whatever they are given. DeScarGAN responds by removing the top skip connection, DRÆM by training a discriminator on synthetic defects. Our base-relative difference is a third answer suited to the continual setting. The shared base network already captures generic image structure, so subtracting its error leaves only what is specific to each task.
Concretely, let $\tilde{x} = C(x; m, \epsilon, t)$ be the input $x$ corrupted by checkerboard mask $m$ and Gaussian noise $\epsilon$ at diffusion level $t$. Let $D_0$ be the frozen base denoiser and $D_k$ the same denoiser with task $k$'s adapter switched on. Let $d$ be a perceptual distance (LPIPS) between an image and its restoration. The routing score is
$$s_k(x) = \frac{1}{J}\sum_{j=1}^{J}\Big[\log d\big(x, D_k(\tilde{x}_j)\big) - \log d\big(x, D_0(\tilde{x}_j)\big)\Big], \qquad \hat{t} = \arg\min_k s_k(x),$$
where the $J$ corruptions $\tilde{x}_j$ are drawn once per test image and reused for every task $k$. That reuse ("common random numbers") is what makes the scores comparable. Variance from the particular mask and noise is shared across tasks and cancels in the argmin, and the log-difference removes per-image difficulty.
Research Program
The base denoiser is a small U-Net diffusion model. In the main from-scratch regime it is trained on task 1's data only and then frozen. Each later task $k$ trains a low-rank adapter, a small set of extra weights added to the frozen network, on task $k$'s images with the standard denoising objective. Adapters are never updated after their task ends. The router therefore cannot forget, and capacity grows by a fixed small amount per task. This avoids SurpriseNet's equal-pruning ceiling, where each task gets $1/T$ of one network.
Because task 1 has no adapter, it is scored by the base alone, with its $s_1$ fixed to the mean self-score on held-out task-1 data. Asymmetries this introduces are reported separately. A second, pretrained-base regime uses a denoiser trained on an external dataset. This matches the pretrained-backbone conditions of the strongest baselines, so the comparison with them is fair.
Within-task prediction is kept deliberately conventional. It is handled by a HAT-masked classifier (hard attention masks that freeze each task's sub-network), as in the Kim et al. stitched systems. Any gain in final accuracy can then be attributed to the router. Following the decomposition, task-ID accuracy and per-task OOD AUC are reported alongside final CIL accuracy.
Inference cost is a first-class quantity. With $J$ corruptions, $L$ noise levels and $T$ tasks, routing costs $J \cdot L \cdot (T+1)$ single network evaluations, with no iterative sampling. We sweep $J \in \{1,4,16\}$ and $L \in \{1,3\}$ to trace an accuracy-versus-compute curve. Multiple adapters can be batched in one forward pass.
The analysis also tests the mechanism, not just accuracy. For raw residuals and for base-relative residuals, we measure the correlation between each image's score and its compressed file size, a proxy for image complexity. If the subtraction works as claimed, the raw score should correlate strongly with complexity and the paired score should not.
Baselines
The naive reference point is reconstruction routing as it exists today. We rerun SurpriseNet's per-task autoencoders on raw pixels and, separately, on frozen ResNet18 features, on the same splits. Its paper reports 16.49% and 48.15% on Split-CIFAR100. Clearly beating the pixel version only confirms that the shortcut problem is real. Beating SurpriseNetE without external features would be the first evidence that pixel-level reconstruction routing can work.
The strongest exemplar-free competitor in the from-scratch regime is HAT+CSI+c from A Theoretical Study on Solving Continual Learning. It reports 65.2% average accuracy on CIFAR-100 with 10 tasks and 47.6% on Tiny-ImageNet with 10 tasks, and its calibration needs a small buffer. It is the right target because it shares our decomposition and our within-task classifier family, and differs only in the OOD detector. We will reuse its HAT classifier and its splits, and give our router the same tuning budget, selected on a validation slice of the current task only. Matching it without the buffer, or exceeding it, counts as a win.
In the pretrained regime the strongest competitors are EWT, which routes by Mahalanobis distance to task-level Gaussians in a pretrained transformer's feature space and reports 72.98% overall average accuracy, and MORE, which reports 70.23% on CIFAR-100 with 10 tasks but uses a replay buffer. We will rerun EWT's Mahalanobis router on our own frozen denoiser's internal features as well as on its original backbone, so the backbone is held fixed.
Four ablations isolate the key ingredient. The first uses raw residuals with no base subtraction. The second subtracts the base but draws fresh noise per task, which isolates the pairing. The third trains independent full-size diffusion models per task, which isolates the shared base. The fourth swaps LPIPS for MSMA-style multi-level denoising-error vectors fed to a per-task Gaussian mixture.
Why not just…?
Why not just use Mahalanobis distance on features, as EWT does? This is the real threat. In feature space it is cheap and strong. EWT's 72.98% exceeds anything reconstruction routing has reported. Its Gaussian assumption is a stated weakness: Cai et al. show that tasks built from arbitrary class groupings are multimodal. However, no one has shown that this costs much in practice. We settle it directly by running Mahalanobis on the very denoiser whose residuals we use. If Mahalanobis on those features matches the paired residual, the reconstruction step adds nothing and we will say so.
Why not just calibrate raw residuals with a small buffer, like +c? The buffer might recover most of the gap the pairing is meant to close. We run raw residuals plus +c calibration as an explicit arm. The claim survives only if the buffer-free paired score matches or beats it.
Why not just reconstruct in a pretrained feature space, as SurpriseNetE does? That gets to 48.15% but inherits whatever the external backbone covers. The original paper still sits far below HAT+CSI+c. Our pretrained-base regime is the fair version of this alternative, and the from-scratch regime tests whether external features are needed at all.
Why not just use the diffusion model for generative replay, as DDGR does? DDGR reports about 75,000 seconds of training on CIFAR-100, against about 8,000 for regularization methods. It also keeps a single shared output head that remains exposed to recency bias, the tendency to favour recently learned classes. We use the generator only to score inputs, with no synthesis loop, and we report wall-clock time so the cost comparison is explicit.
Proposed Experiments
We use the standard splits from the file: CIFAR-10 in 5 tasks, CIFAR-100 in 10 and 20 tasks, and Tiny-ImageNet in 10 tasks. We report CIL average accuracy, task-ID accuracy, per-task OOD AUC, network evaluations per test image, and added parameters per task, each over three seeds and task orderings.
The decisive check is on CIFAR-100 with 10 tasks, from scratch. The paired router with a HAT classifier must outperform both raw residuals and Mahalanobis on the same denoiser's features. It must also approach or beat 65.2% without a buffer. The complexity-correlation diagnostic must show the mechanism: high correlation for raw residuals and near-zero for paired ones.
A second check tracks how the margin over Mahalanobis changes with task count, from 10 to 20 tasks. Smaller tasks make clusters more irregular, which is where a manifold-based score should help most. If the paired router loses to Mahalanobis on its own features in both checks, the conclusion is that reconstruction is the wrong routing signal for near-OOD tasks, and that negative result is worth publishing.
References
- A Theoretical Study on Solving Continual Learning
- Look At Me, No Replay! SurpriseNet: Anomaly Detection Inspired Class Incremental Learning
- Unsupervised Out-of-Distribution Detection with Diffusion Inpainting
- Multiscale Score Matching for Out-of-Distribution Detection
- DRÆM - A discriminatively trained reconstruction embedding for surface anomaly detection
- DeScarGAN: Disease-Specific Anomaly Detection with Weak Supervision
- Solving Continual Learning via Problem Decomposition
- A Multi-Head Model for Continual Learning via Out-of-Distribution Replay
- Multi-Centroid Task Descriptor for Dynamic Class Incremental Inference
- DDGR: Continual Learning with Deep Diffusion-based Generative Replay