Bridging Task Shifts by Sliding Feature Statistics: A Cheap Domain Flow for Forward Transfer

When a continual learner, meaning a model trained on a sequence of tasks, moves from one task to the next, it should learn the new task faster than a fresh model would. Often it does not. On streams where consecutive tasks share content but differ in appearance, such as the same digits rotated or the same objects under a new corruption, the first gradient steps on the new data can be large and distorting. They overwrite features that would have transferred, so the model learns no faster than a fresh one, or slower. We propose a small change to how each new task begins. For a short period, the learner sees the new task's data with its internal feature statistics (the per-channel means and variances that roughly encode visual "style") pulled back toward those of the previous task. A single scalar fader then slides those statistics toward the new task's own values. The fader advances only when the model is coping, and it then disappears.

The idea borrows from domain flow, where a translation network generates a continuum of intermediate domains between a source and a target, and moves it into the learner's own normalization layers. There it costs a few stored vectors per task instead of a separately trained image generator. The claim is specific. On domain-incremental streams, gradual statistic interpolation will raise forward transfer, measured as early-learning speed and accuracy on each new task, beyond what replay, invariance penalties, relaxed gradient projection, or simple step-size warmup achieve. The gain should grow with the size of the style shift and vanish when there is no style shift.

Core Claim

Forward transfer fails on appearance-shift streams partly because the new task arrives as a sudden distribution shock, not because the old features are useless. If that is right, then giving the network a path of intermediate distributions, each close to what it already handles, should let it adapt by small corrections that keep the reusable features intact. The path is built from the new task's own samples, so labels stay correct and content stays realistic. Only first- and second-order feature statistics are moved.

Concretely, let $\mu^{l}_{t-1}, \sigma^{l}_{t-1}$ be the stored channel means and standard deviations of layer $l$ on the previous task. Let $\mu^{l}(h), \sigma^{l}(h)$ be the statistics of the current feature map $h$ computed on the new task's batch. During the bridge phase, each chosen layer's features are re-normalized as

$$ \tilde h = \big[(1-\alpha)\sigma^{l}_{t-1} + \alpha\,\sigma^{l}(h)\big]\,\frac{h-\mu^{l}(h)}{\sigma^{l}(h)} + (1-\alpha)\mu^{l}_{t-1} + \alpha\,\mu^{l}(h). $$

Here $\alpha \in [0,1]$ is the fader. At $\alpha=0$ the new data is rendered in the old task's statistics. At $\alpha=1$ the operation is the identity. The fader advances by a fixed increment whenever the running training loss at the current $\alpha$ falls below a threshold. The schedule is therefore self-paced rather than fixed. Once $\alpha=1$, training is ordinary.

Relation to Prior Work

The direct motivation comes from two benchmark papers. CLiMB evaluates continual vision-and-language models. It finds that standard methods, including experience replay, transfer negatively: prior training made the model 5.09% worse at learning VCR (Visual Commonsense Reasoning) than training from scratch. Continual World is a robotic manipulation benchmark. It reports that most continual methods score near zero or negative on its forward transfer measure. Both establish the deficiency. Neither isolates abrupt distribution change as a cause.

The mechanism is borrowed from the latent style interpolation line. DLOW builds a CycleGAN-style translator conditioned on a scalar "domainness" variable through conditional instance normalization. It uses the generated intermediate domains to raise GTA5→Cityscapes mIoU (mean intersection over union) from 36.6% to 42.3%. We keep the scalar fader and the normalization-statistics view of style. We drop the generator and the two-discriminator training, which the file flags as too heavy for streams. We also apply the flow at every task boundary of a $T$-task stream rather than once.

Fader Networks slide a continuous attribute input into a decoder after adversarially removing that attribute from the latent code. That design runs into an invariance-versus-content trade-off, which matters here because forward transfer needs content kept. Our fader never strips anything from the representation. It only shifts statistics temporarily, which sidesteps that tension.

Several methods target forward transfer through parameter space. ROGO relaxes orthogonal gradient projection to allow oblique updates. CKA-RL composes new policies from stored parameter offset vectors. Transferability-aware task embeddings condition a hypernetwork on a 32-dimensional transferability embedding. CLeWI interpolates weights backward toward the previous model after training, to reduce forgetting. All of these change how parameters move given an abrupt data shift. We instead change the data path the parameters experience, and we do it prospectively at the start of the task rather than retrospectively. The two are compatible, and one experiment stacks ours on ROGO.

The closest method on domain shift is ★-CL, which adds invariance penalties such as CORAL to replay. CORAL also works on second-order feature statistics, but as a permanent penalty that forces representations to look alike across domains. We impose no penalty. We temporarily move the input's statistics and then release them, so representations remain free to become domain-specific where that helps. Shared and Private VAEs use generative replay to prevent forgetting. That constrains our design: synthesizing whole samples is effective but grows per task, which is why we interpolate statistics rather than generate images.

Research Program

We will build the bridge as a drop-in module for batch-normalized or instance-normalized convolutional networks, starting with ResNet-18. At the end of each task it stores per-layer channel statistics, a few kilobytes. We will study three design questions. The first is which layers to bridge: early layers only, matching the intuition that appearance lives low in the network, versus all layers. The second is whether a self-paced schedule beats a fixed linear one. The third is how the bridge interacts with replay, since in practice it would be combined with a small buffer.

For mechanism, we will track two quantities. One is the norm of early gradients on the new task. The other is feature reuse, measured with linear CKA (a similarity score between two sets of layer activations) between each layer's features before and after the new task. The hypothesis predicts lower early gradient norms and higher CKA on early layers, together with faster new-task learning.

The setting is domain-incremental with a single shared output head. No task identity is needed at test time, which avoids the oracle-ID dependence of modular methods such as MNTDP. The bridge needs to know when a task boundary occurs. In a task-free variant, a jump in the running batch statistics triggers it.

Baselines

The naive reference point is sequential fine-tuning with no preservation mechanism and no bridge. It sets the forward-transfer level that any method must beat. Continual World reports fine-tuning at a forward transfer of 0.20 against a reference ceiling of 0.46, which shows that fine-tuning is a nontrivial transfer baseline rather than a straw man. A from-scratch learner on each task defines zero transfer.

Experience replay with a 500-sample buffer is the standard practical method. On five-task CIFAR-10, CLeWI reports 53.22% average accuracy for replay alone and 62.8% with CLeWI. We include ER+CLeWI as the strongest interpolation-based method, on the same buffer.

The strongest existing method for the domain-shift streams we target is ★-CL with its CORAL or MMD variants. The file reports only relative gains for it: 2 percentage points over replay baselines, 6 over optimization-based and 10 over regularization-based methods. No absolute figure on RotatedMNIST, CIFAR-10-C or TinyImageNet-C was found on Lacuna, so we will rerun it ourselves.

ROGO is the strongest method that explicitly targets forward transfer. It reports $\Omega_{new}$ of 63.81% on MiniImageNet against 61.63% for GPM, and 74.04% versus 72.48% split accuracy on CIFAR-100. On the CTrL input-shift stream we will also run MNTDP. The file reports 75% for MNTDP on the 100-task long stream, but no reported result was found on Lacuna for its input-shift stream.

All methods share the same backbone, data order, epochs per task, buffer size, and a matched hyperparameter search budget of 20 trials per method, run on a held-out validation stream. Because the self-paced bridge uses extra steps at intermediate $\alpha$ values, baselines receive the same total number of gradient steps. Beating a baseline means a higher area under the new-task learning curve and higher final accuracy, with forgetting no worse, at matched compute, over five seeds.

The key ablations isolate the ingredient. A fixed-schedule bridge tests whether self-pacing matters. A bridge that interpolates toward random statistics rather than the stored ones tests whether the old task's statistics specifically matter, or whether any noise injection would do. Finally, ER+bridge and ROGO+bridge test whether the gains are additive with existing methods.

Why not just…?

Why not just mix old and new images with mixup? Pixel-space blending also creates intermediate distributions, and it is the cheapest competitor. Mixed images have blended labels and unrealistic content, while our bridge keeps each new-task sample's content and label and moves only its statistics. Whether that difference matters is empirical. We run buffer-based mixup with a schedule that anneals the old-sample weight from high to zero, using the same pacing rule as our bridge. If it matches statistic interpolation, the contribution reduces to "curricula at task boundaries help," and we would report that.

Why not just warm up the learning rate, or linear-probe then fine-tune? This is the most serious threat. If the deficiency is simply large early gradients, small early steps or training only the head first should fix it without touching the data. We include both, tuned on the same budget. The distinguishing prediction is the shift-magnitude curve. Warmup should help roughly uniformly. The bridge should help more as rotation angle or corruption severity grows, because it reduces the shift itself rather than only the step size. If warmup matches the bridge across all shift magnitudes, the hypothesis that abrupt distribution change causes the deficiency is wrong, and we would say so.

Why not just use a DLOW-style image generator, which is known to work? It might work better, and on the smaller streams we will run it as an upper reference, training a CIN-conditioned translator per boundary. Its cost scales with the number of task boundaries and the image resolution. Our question is how much of its benefit survives when the generator is replaced by a few stored statistics. We will report compute alongside accuracy.

Why not just add an invariance penalty such as CORAL? That is what ★-CL does, and it is a baseline. A permanent penalty pushes representations to be identical across domains. The tension the Fader Networks literature documents between invariance and retained content suggests this can erase useful domain-specific features. Our layer-wise CKA measurements, together with final new-task accuracy, will show whether a temporary bridge keeps more transferable structure than a permanent penalty does.

Proposed Experiments

The primary benchmarks are RotatedMNIST, CIFAR-10-C and TinyImageNet-C. Each is arranged as a stream of domains (rotation angles, or corruption types at fixed severity) with shared classes, following the ★-CL protocol, and the CTrL input-distribution-shift stream from MNTDP. The metrics are three. First, forward transfer, defined as new-task accuracy after a fixed small step budget relative to a from-scratch learner. Second, area under the new-task learning curve. Third, final average accuracy and forgetting.

Split CIFAR-100 serves as a negative control. Its tasks are new classes, not new styles, so the hypothesis predicts little or no gain there. A large gain on it would mean the bridge is acting as a generic regularizer rather than a shift reducer.

Two checks decide whether the claim holds. The first sweeps rotation angle on RotatedMNIST and severity on CIFAR-10-C. The hypothesis predicts that the bridge's advantage over warmup and mixup increases with shift size, and that at small shifts its advantage over sequential fine-tuning shrinks toward zero. The second is mechanistic: compared with fine-tuning, the bridge should show higher early-layer CKA to pre-task features and lower early gradient norms. If forward transfer improves without these signatures, our explanation is wrong even if the method works. If the signatures appear without a transfer gain, the shock hypothesis does not explain the deficiency.

As a stretch test, we will apply the same bridge to the LayerNorm statistics of a ViT with LoRA adapters on ImageNet-R. The file's hypernetwork method reports 81.38% final average accuracy there against SD-LoRA's 77.18%. This tests whether the effect carries over beyond convolutional networks.