Expanding Only at Discovery Significance: Calibrated Growth Tests for Modular Continual Learners

Continual learners that never forget usually work by freezing a shared network and adding a small new module, such as an adapter, every time something new arrives. The cost is that memory grows in step with the number of tasks. The best current fix is to add a module only when the incoming data looks unfamiliar. That decision is a novelty detector with a hand-set threshold, and nobody controls how often it fires on data that is not actually new. This proposal treats the decision to grow as a hypothesis test, borrowed from rare-event physics. Ordinary, already-familiar data plays the role of "background." A genuinely new distribution plays the role of "signal." The model may add parameters only when the evidence for signal reaches a pre-declared significance, with a provable cap on how often that happens by chance.

The payoff is a statement current methods cannot make. If the detector is calibrated against held-out data from the modules the model already has, then the number of false expansions over an entire stream is bounded by a user-chosen error budget. Total parameter growth is then tied to the number of truly new distributions, not the number of tasks or batches. The same calibrated scores also give the deployed model a principled way to abstain on inputs that no existing module covers.

Core Claim

Self-expanding continual learners already exist. SEMA attaches a small autoencoder, which it calls a "representation descriptor," to each adapter. It adds a new adapter only when the reconstruction-error z-score of incoming features exceeds a threshold, and otherwise reuses frozen adapters through a soft router. CoPE uses temporary probe experts to decide which layers of a vision-language model get new low-rank experts. Both decide when to grow with a manually tuned threshold. In both, the false-alarm rate of that decision is unknown and depends on stream length, window size and score noise. The research file flags exactly this weakness: in a stream dominated by familiar data, reconstruction z-scores fire on ordinary fluctuations and trigger spurious growth.

Our claim is that the growth decision should be the object of error control. We define a window statistic $S_t$, such as mean novelty score over a window of $w$ incoming samples, and compare it against a reference set $\{S_j\}_{j=1}^n$. The reference set is computed from held-out windows of the data each existing module was trained on, which is the analogue of a physics sideband. This gives a rank-based p-value

$$p_t = \frac{1 + \#\{j : S_j \ge S_t\}}{n+1},$$

which is valid whenever the incoming window is exchangeable with the reference windows, meaning drawn the same way from an already-covered distribution. The model expands at step $t$ only if $p_t \le \alpha_t$. The per-test levels follow an alpha-spending schedule with $\sum_t \alpha_t \le \alpha$. A union bound then gives that, with probability at least $1-\alpha$, no expansion is ever triggered by familiar data over the whole stream. This is the continual-learning counterpart of correcting for the "look-elsewhere effect," where scanning many places inflates chance discoveries. On that event, the number of modules equals at most the number of genuinely new distributions detected.

This differs from SEMA and CoPE in what is controlled, not in architecture. We keep SEMA's adapters and router and change only the expansion rule and the inference-time abstention rule.

Why the Four Directions Meet Here

The physics literature supplies the method. CATHODE on CMS Open Data fits a background model on sidebands outside the signal window and classifies data against it. It lifted an Upsilon resonance from 1.6σ to 6.4σ, and it validated the absence of artificial peaks with a same-sign control in which no signal should appear. The LHC Olympics 2020 motivates the need for such controls: methods were tested on a background-only black box, and a complex multi-mode signal went mostly undetected. We import three pieces: a sideband-calibrated null, a mandatory background-only control stream, and a stated significance threshold.

Selective prediction supplies the guarantee style. Selective Classification for Deep Neural Networks shows how to choose a confidence threshold on held-out data so that error among accepted inputs stays below a target $r^*$ with probability $1-\delta$. That result assumes a frozen predictor. We apply the same logic twice. During training, "reuse an old module" is the prediction and "expand" is the abstention. At inference, inputs whose p-value against every module's reference set is small are deferred rather than routed.

The open-set trade-off is the failure we must not reintroduce. Fixed-capacity novelty methods such as SHELS pay for rejection with in-distribution accuracy and eventually run out of free capacity. Generalized ODIN likewise trades a little top-1 accuracy for detection. In our setting the trade-off has two sides. A threshold that is too lax absorbs a new distribution into an old adapter and hurts accuracy. A threshold that is too strict spawns redundant adapters, which costs memory and confuses routing. A calibrated test lets us place the operating point explicitly rather than by tuning.

Research Program

We first build the test on top of a reimplementation of SEMA's adapters and representation descriptors. Each module stores a small reference set of held-out window statistics, a few hundred scalars, which is negligible next to an adapter. We study three choices of novelty score:

The test wraps any of these. We also compare two decision rules: a single fixed window, and a sequential rule that accumulates evidence across windows. The trade-off between them is detection delay versus false alarms.

Second, we characterise when validity breaks. Exchangeability fails under slow within-task drift, which is the continual-learning analogue of background mismodeling. We will measure the empirical false-expansion rate on drifting but semantically unchanged streams. If it exceeds $\alpha$, we will test the cheapest repair, which is periodically refreshing each module's reference set from recently accepted data. Without a guard, refreshing can absorb real novelty, and we will report that risk honestly.

Third, at inference, we use stored per-module reference sets to compute routing p-values and apply an SGR-style threshold search. The goal is a target selective risk on seen tasks, with rejection reserved for inputs no module covers.

Baselines

The naive reference points are the two extremes. One adds an adapter for every task, which gives linear growth and is roughly an accuracy upper bound for isolation methods. The other never expands, with a single adapter trained once, which gives a floor. Any proposal must sit between these on accuracy while staying close to the floor on parameters.

The strongest existing competitor is SEMA itself, because it is the method designed to avoid linear growth through novelty-triggered expansion. The file reports SEMA at 64.53% average accuracy on ImageNet-A versus 47.29% for CODA-P, per SEMA. No reported result was found on Lacuna for SEMA's adapter counts, or for its CIFAR-100, ImageNet-R and VTAB numbers, so we will re-measure all of these. For fairness, we run SEMA with the same backbone, adapters, data order and training budget. Its z-threshold receives the same number of tuning trials we spend on choosing window size and $\alpha$, using the same held-out tuning stream.

Beating SEMA means three things. First, accuracy within one point of SEMA on its benchmarks. Second, strictly fewer adapters. Third, the decisive criterion: a false-expansion rate at or below $\alpha$ on null and transfer streams where SEMA's tuned threshold exceeds it.

For the vision-language setting, CoPE on CoIN is the reference. It reports 59.23% average accuracy after eight tasks against 42.41% for plain LoRA, with 43.96M added parameters per task, per CoPE. We would replace CoPE's probe-activity threshold with our test and compare added parameters at matched accuracy.

The key ablation keeps everything fixed and swaps only the decision rule: raw thresholded score versus calibrated p-value with alpha-spending. This isolates calibration and multiple-testing control from the choice of score.

Why not just…?

Why not just tune SEMA's threshold more carefully? This is the real threat. On a single benchmark, a well-tuned threshold may match our method exactly. The claim is about transfer. A threshold tuned on one stream has no guarantee on a longer stream, a different window size, or a stream with more repeated tasks, because the chance of a spurious crossing grows with the number of tests. We settle this by tuning SEMA's threshold on CIFAR-100 splits and freezing it on two kinds of stream. The first is ImageNet-R and VTAB. The second is a null stream that re-presents already-learned tasks. If SEMA's frozen threshold keeps its false-expansion rate as low as ours across these, our contribution reduces to a cleaner tuning procedure, and we will say so.

Why not just use a strong closed-set classifier's logits as the novelty signal? Vaze et al. show that maximum logit score from a well-trained classifier rivals specialised open-set methods. We accept that, and the maximum logit score is one of our candidate scores. It does not answer when to grow, however. Any score still needs a threshold whose false-alarm rate is controlled across hundreds of tests. Vaze et al. also report that the approach weakens on semantically close unknowns. We therefore include fine-grained task orders, where adjacent tasks are similar, to see whether detection delay explodes.

Why not just apply per-sample conformal p-values with a Bonferroni correction? This is close to our method, and it is the honest nearest alternative. Per-sample tests at stream scale need tiny per-test levels and lose almost all power when new data arrives diluted among familiar data. That is the low signal-to-background regime the physics literature handles by aggregating evidence in windows. We will run per-sample Bonferroni as a baseline. If it detects new tasks with comparable delay, windowing adds nothing and we will drop it.

Proposed Experiments

We use the class-incremental benchmarks from SEMA (CIFAR-100, ImageNet-R, ImageNet-A, VTAB) and CoIN from CoPE. We add two constructed streams that mirror the physics controls.

The first is a background-only stream, the analogue of the LHC Olympics null black box and CATHODE's same-sign check. Already-learned tasks are re-presented in shuffled order. The correct number of expansions is zero, and we report the fraction of 100 random seeds with any expansion against the nominal $\alpha$.

The second is a dilution stream. Data from a new task enters mixed into familiar data at fractions from 50% down to 1%. We report detection delay and missed detections as a function of that fraction, which is the continual-learning version of a signal-to-background scan.

Metrics are average and final accuracy, backward transfer, number of adapters against the true number of distinct distributions, false-expansion rate, and detection delay. At inference we add risk–coverage curves with the realised risk checked against the target. Selective Classification for Deep Neural Networks reports 1% error at 78% coverage on CIFAR-10 for a frozen model, which serves as a reference for what guaranteed selection costs in coverage.

The decisive checks are three. The empirical false-expansion rate on null streams must stay at or below $\alpha$ while tuned-threshold SEMA exceeds it after transfer. Adapter count must track true distribution count rather than task count. Accuracy must not fall below SEMA's. If the first holds but the third fails, calibration is buying memory at the cost of accuracy, and the open-set trade-off has simply moved rather than been managed.