Certified Background Templates for Exemplar-Free Task Prediction

A continual classifier learns classes a few at a time and must later label any test image among all of them, without being told which training stage it came from. Such models often keep good features for old classes and still fail, because they never learned to separate an old class from a new one. That failure is called inter-task confusion. The strongest known fix, TPL, predicts which task an input belongs to with a likelihood ratio: "how typical is this input of task $t$" divided by "how typical is it of all the other tasks." The bottom of that ratio is estimated from stored raw images. Drop the stored images and the denominator must come from a learned summary of old data, which particle physicists would call a background template. Physicists have learned the hard way that a classifier asked to separate real data from an imperfect synthetic template will often learn to spot "synthetic" rather than "signal." They have built null-control tests to catch this before trusting the result.

This note proposes carrying that discipline into continual learning. The claim is concrete. In exemplar-free continual learning, the familiar bias toward predicting the newest task is partly a template-mismatch artifact. The new task is represented by real features and the old tasks by synthetic ones, so a task predictor can win by detecting realness. The proposal makes three moves. It measures this mismatch with a null-control test while the real data still exists. It removes the artifact by training the task predictor only on templates, for every task including the current one. It then tests the method in an asymmetric regime, where one huge, drifting background must be separated from rare signal classes that arrive one at a time. That is the regime of rare-event searches, and no continual learning benchmark covers it.

Core Claim

Write $s_t(x)$ for the output of a small classifier trained to tell task-$t$ features (label 1) from features of all other seen tasks (label 0). Under balanced training, this output gives the task likelihood ratio through the standard classifier-to-ratio identity:

$$\Lambda_t(x) = \frac{p_t(x)}{p_{t^c}(x)} = \frac{s_t(x)}{1 - s_t(x)} .$$

Here $p_t$ is the feature density of task $t$ and $p_{t^c}$ is the density of its complement. The identity only holds if both sides of the training set are draws from the true densities. Exemplar-free methods break this assumption. The current task is real, and old tasks come from a template $q_{t^c}$ such as a per-class Gaussian, a normalizing flow, or inverted images. The classifier then estimates $p_t / q_{t^c}$, and any direction in which $q_{t^c}$ differs from $p_{t^c}$ becomes a free cue. We predict that this cue makes old-task test inputs, which are real, look like the newest task. That would show up as the recency bias usually blamed on drifting weights.

CATHODE faces the same issue. It trains a normalizing flow, a generative model that maps data to a simple distribution invertibly, on sideband regions where the signal is absent. It interpolates that flow into the signal region and trains boosted trees to separate data from the synthetic template. It guards against fake peaks with a same-sign dimuon control sample in which no signal can exist. We borrow both halves. The template-then-classifier construction supplies the method. The null control becomes a quantitative diagnostic: a two-sample classifier AUC between held-out real features of a task and that task's template, computed before the real data is discarded. If our account is right, this "mismatch AUC" should predict the size of recency bias across template families and benchmarks. Training the task predictor on templates for all tasks, so that real-versus-synthetic cues appear on both sides and cancel, should shrink that bias.

Relation to Prior Work

Three papers motivate the gap. The mathematical framework of Khademi Nori et al. and the first-principles review show that discriminative training never constrains boundaries between classes from different tasks. That is why task prediction is the bottleneck. Joint input and output coordination and OVOR document that old inputs fire new heads strongly, a calibration interference our diagnostic should partly explain.

Two papers supply the decomposition we reuse. CLOM splits continual classification into per-task classifiers (task-incremental learning) plus out-of-distribution gating. TPL sharpens the gating into a Neyman-Pearson ratio whose complement is the other seen tasks. TPL also reports that task-detection AUC tracks final accuracy linearly. That makes task-ID AUC the right intermediate quantity to study. We keep TPL's decomposition but replace its replay-buffer $k$-nearest-neighbor denominator with a template-trained classifier. We do not claim the ratio idea itself.

Exemplar-free and compressed rivals are the comparison class. Metric learning with a reused past model stores class centroids as negative anchors. AHR replays compressed latents and pushes centroids apart with a heuristic repulsion. BEEF uses energy prototypes and sampling to reserve space for past and future classes. All three build a summary of old data, but none checks whether the summary is faithful before relying on it. Our diagnostic applies directly to their stored summaries.

On the physics side, the HPGe alpha-rejection work and the sPHENIX real-time trigger fix the evaluation language for the asymmetric regime: signal kept at a fixed, very small background acceptance. The HPGe work rejects surface alphas while losing only 0.2% of bulk events. The sPHENIX work also constrains the design. A deployable task predictor must be a small fixed network, not a nearest-neighbor search, which favors our template-trained classifier over a buffer search.

Research Program

The first stage is measurement in a clean setting. We use a frozen pretrained transformer, as in TPL's pretrained configuration, so that all tasks share one feature space and template quality is the only moving part. For each task we fit three template families of increasing capacity on its features: per-class Gaussians, Gaussian mixtures, and a small normalizing flow. On held-out real features of the same task we record the mismatch AUC. We then train task predictors two ways. The asymmetric way uses real current-task features against templates of old tasks, which is what a naive exemplar-free method does. The symmetric way fits the current task's template first and trains on templates only. We measure task-ID AUC, final accuracy, and recency bias, meaning the fraction of old-task test inputs assigned to the newest task.

The second stage lets the backbone adapt through per-task masks, as in CLOM and TPL. Old templates then go stale as features drift, which is the continual-learning analogue of changing run conditions. We test whether re-estimating the mismatch AUC on fresh templates of the current task can flag drift without old data. We also test whether template refreshing through the stored mask-specific heads limits it.

The third stage builds the asymmetric benchmark. One "background task" draws from a large, broad pool of classes whose composition shifts over time. A sequence of "signal tasks" each contributes a single rare class, injected into the test stream at rates of $10^{-3}$ to $10^{-4}$. The model must keep one background template current while adding signal classes. We report signal efficiency at fixed background acceptance, alongside accuracy.

A final bounded transfer check returns the method to physics. On the 2016 CMS DoubleMu open data, we treat successive dimuon mass windows as sequential tasks. The flow fitted for each window's sidebands is retained only as a template, and we run the same-sign control sample through the sequential version as the sculpting check.

Baselines

The naive reference point is a shared single-head classifier fine-tuned task by task, with no replay and no task prediction. It sets the size of inter-task confusion. Joint training on all data is the upper bound.

The strongest existing method is TPL. It reports an average Last accuracy of 76.21% across its five benchmarks, against 73.72% for the best prior baseline (ROW). The file gives no per-dataset TPL numbers, and we found no reported result on Lacuna for TPL on ImageNet-380 alone, so we rerun TPL ourselves. We use TPL's backbone, masks, and tuning protocol. Its buffer is set to the same number of bytes our templates occupy, and also to its native size. Beating TPL means matching or exceeding its task-ID AUC and Last accuracy at matched memory while storing no raw inputs. Matching it at native buffer size would be a strong result.

The strongest exemplar-free competitor is the metric learning method. It reports 54.8% average accuracy on 10-task CIFAR-100 and 54.1% on 20-task TinyImageNet, so it is the bar on those exact splits. CLOM reports 47.6% class-incremental accuracy on Tiny-ImageNet-10T. AHR reports 54.43% on CIFAR-100. We will check that our reruns reproduce these numbers before any comparison is reported.

The key ablation isolates the proposal's one new ingredient. It holds template family, backbone, and classifier fixed and changes only whether training is asymmetric (real current versus synthetic old) or symmetric (synthetic everywhere). A second ablation replaces the classifier ratio with the direct density ratio of the templates. That tests whether the classifier step adds anything beyond the template itself. All methods share data order, backbone, three seeds, and an equal hyperparameter search budget.

Why not just…?

Why not just keep a small raw buffer and use TPL? This is the real threat. If a few hundred stored images beat any template at equal bytes, the exemplar-free setting holds up only where privacy or the buffer costs flagged for TPL and AHR forbid storage. We will report the full accuracy-versus-bytes curve for both methods and state plainly where the curves cross. The diagnostic contribution stands even if TPL wins on accuracy at large memory.

Why not just use per-class Gaussians, which are cheap and already common? They may be enough. Our claim concerns how templates are used, not which family is best, so Gaussians appear as the first template family. If symmetric training with Gaussians closes most of the gap, the flow is unnecessary and we will say so. The mismatch AUC tells us in advance whether a family is good enough, which is the practical payoff.

Why not just recalibrate logit scales across tasks, as BEEF's affine fusion does? Rescaling corrects a global offset between heads. Template mismatch is a direction-specific cue that varies with the input, so a single scale cannot remove it. The decisive check is to apply the best per-task affine recalibration to the asymmetric predictor. If recency bias and task-ID AUC then match the symmetric predictor, our mechanism is wrong.

Why not just retrain from pooled data offline? That is the joint upper bound, and we report it. It requires keeping all past data, which the continual setting excludes. In streaming triggers like sPHENIX, unrecorded data is lost permanently, so there is nothing to pool.

Decisive Checks

The idea succeeds or fails on three outcomes. First, across template families and the CIFAR-100, TinyImageNet, and ImageNet-380 splits, the mismatch AUC measured at training time should correlate with recency bias at test time. If no correlation appears, the central explanation is wrong. Second, symmetric training should reduce recency bias and raise task-ID AUC relative to asymmetric training, with everything else fixed. If it does not, the realness cue is not what drives the bias. Third, in the asymmetric background-plus-rare-signal benchmark, the template-trained predictor should hold signal efficiency at $10^{-4}$ background acceptance as the background drifts. TPL with a matched-byte buffer should lose efficiency as its fixed sample of the background goes stale.

For the CMS transfer, the bar is that the sequential version shows no significant excess in the same-sign control sample. It should also approach the 6.4σ Υ significance CATHODE reports offline, up from 1.6σ for the raw spectrum. Falling well short there would mark the limit of the transfer, not a failure of the core claim.