Continual learners are almost always tested on tidy task streams in which every task gets the same number of training steps. Real streams are not like that. A deployed model may sit in one regime for weeks, see a dozen short shifts in an afternoon, then return to a long regime. This proposal asks whether a learner's ability to keep old knowledge and pick up new knowledge depends only on how much training it receives on average, or also on how unevenly that training is spread across tasks. Queueing theory has a name for the first property, distributional insensitivity: a system's average performance depends only on mean load, not on the variance or tail of job sizes. It also has a recipe for achieving it: share capacity among active jobs the way a processor-sharing server does, rather than letting the first large job hold the resource.
The idea in two sentences: uncertainty-guided continual learners, which give each weight a learning rate proportional to its posterior variance, behave like a first-come-first-served queue, because one long task drives the variances it touches toward zero and freezes the network for every short task after it. We propose to (i) measure this sensitivity directly by holding the mean task duration fixed while varying its distribution, and (ii) fix it with a "balanced precision" rule that gives each task context an equal, duration-independent share of a fixed consolidation budget.
Core Claim
In a Bayesian continual learner such as Uncertainty-guided Continual Bayesian Networks, or UCB (Uncertainty-guided Continual Learning with Bayesian Neural Networks), the mean of each weight moves by $\Delta\mu_i \propto \sigma_i^2 g_i$, where $\sigma_i^2$ is that weight's posterior variance and $g_i$ its gradient. Precision $\tau_i = 1/\sigma_i^2$ accumulates roughly linearly with the number of examples seen. A task of duration $d_k$ therefore contributes consolidation proportional to $d_k$. Under heavy-tailed durations, a few long tasks dominate the total precision. This has two consequences: earlier short tasks are overwritten in proportion to their share, which is small, and later tasks inherit a frozen network.
We claim this makes current learners duration-sensitive. At fixed mean duration and fixed total training steps, their average accuracy, backward transfer, and new-task learning speed should degrade as the duration distribution moves from constant to exponential to Pareto. We predict a dependence on the second moment $\mathbb{E}[d^2]/\mathbb{E}[d]$, the same size-biased quantity that makes first-come-first-served queueing delay depend on job-size variance.
We further claim that a balanced precision rule makes these metrics depend, to first order, only on the number of contexts and the total step budget. The rule has two parts. First, normalize each context's accumulated Fisher information (a per-weight curvature estimate) to a fixed trace. Second, keep the total precision at a constant budget $B$ that is shared equally among the $n$ contexts seen so far:
$$ \tau_i = \tau_0 + \frac{B}{n}\sum_{k=1}^{n} \frac{F^{(k)}_i}{\mathrm{tr}\,F^{(k)}} , $$
Here $\tau_0$ is the prior precision, $F^{(k)}_i$ is the diagonal Fisher accumulated for weight $i$ during context $k$, and $\mathrm{tr}\,F^{(k)}$ is the sum of $F^{(k)}$ over all weights. The per-context normalization is the insensitivity ingredient, playing the role of processor sharing. The fixed budget is the anti-rigidity ingredient: total precision can never diverge, so variances cannot all collapse.
Relation to Prior Work
Two lines of work motivate the proposal. On the queueing side, On performance bounds for balanced fairness shows that capacity allocations satisfying a balance condition make flow delays depend only on mean load and that they dominate store-and-forward allocation. Time-based Fairness Improves Performance in Multi-rate WLANs shows the practical version of the same lesson: allocating channel time rather than packet count stops slow clients from monopolizing a WLAN. We borrow the allocation principle only. We do not claim that the reversibility proofs transfer to learning dynamics; whether a learner is insensitive is an empirical and, for linear models, analytical question we test.
On the learning side, Uncertainty-guided Continual Learning with Bayesian Neural Networks and Task-Agnostic Continual Learning Using Online Variational Bayes With Fixed-Point Updates (FOO-VB) supply the variance-as-learning-rate mechanism we modify. Both accumulate precision in proportion to data seen, and both were evaluated on uniform-length tasks.
Three papers constrain the design. Loss of plasticity in deep continual learning and Spectral Collapse Drives Loss of Plasticity in Deep Continual Learning show that progressive locking kills learnability over long streams. This is why we cap total precision rather than simply reweighting it. BAM: Bayes with Adaptive Memory restores plasticity by unshrinking variance, but it needs a buffer of past data. Our budget rule is memory-free.
Two near neighbours differ in mechanism. MoCL: Metabolic Optimization for Curvature-Aware Continual Learning notes that Fisher spectra are heavy-tailed and gates updates through an entropic function. That paper is about skew within the curvature spectrum, whereas we address skew in task durations. Confidence Based Dual Reinforcement Q-Routing modulates tabular routing updates by confidence inside a network. It never examines insensitivity or neural parameter uncertainty.
Research Program
The first deliverable is a duration-sensitivity protocol. A stream consists of $N$ contexts. Durations are drawn independently from a family that shares the same mean $\bar d$: deterministic, exponential, and Pareto with tail index 2.5 and 1.5. Durations are rescaled so every stream has exactly $N\bar d$ total steps, and each condition uses several random seeds.
For every learner we report three quantities:
- Final average accuracy over all contexts.
- Backward transfer, meaning how much accuracy on earlier contexts drops by the end.
- Plasticity, defined as accuracy on a new context after its first $m$ steps.
Each quantity is plotted as a function of the duration distribution. A sensitivity index summarizes each plot: the spread of the metric across the four distributions, divided by its seed standard deviation. An insensitive learner has an index near one.
The second deliverable is analysis in Bayesian linear regression with diagonal precision. There, forgetting of context $k$ can be written in closed form. We expect to show that its expectation under linear accumulation involves $\mathbb{E}[d^2]$, while under the balanced rule it depends only on $n$ and $B$.
The third deliverable is the balanced precision learner itself, in two forms:
- Task-aware. Context boundaries are given.
- Task-agnostic. Contexts are opened by a change detector, specifically a CUSUM test (a running sum that flags a sustained rise) on the online loss and predictive entropy.
Memory stays at two to three copies of the parameters. When a context closes, its trace-normalized Fisher is folded into the aggregate with weight $B/n$, and the aggregate is rescaled by $(n-1)/n$. No per-context storage is retained. A known weakness of the task-agnostic form is that false change detections split one long task into several contexts, which reintroduces duration sensitivity. We will measure this directly by injecting controlled false-alarm rates.
Baselines
Naive reference. Plain SGD/Adam with tuned weight decay. It anchors how much any method helps, and the plasticity literature shows that weight decay alone only partly slows decay (Loss of plasticity in deep continual learning).
Strongest uncertainty-modulated competitors. UCB in the task-aware setting and diagonal FOO-VB in the task-agnostic setting. These are exactly the methods whose precision accumulation we hypothesize is duration-sensitive. On uniform streams, UCB reports 99.63% multi-head accuracy on Split MNIST and backward transfer of −0.38% on Permuted MNIST, against −7.90% for VCL (Uncertainty-guided Continual Learning with Bayesian Neural Networks). FOO-VB in its matrix-variate form reports less than 1% average-accuracy degradation after 10 tasks of continuous Permuted MNIST, and 88–94% accuracy for the diagonal variant (Task-Agnostic Continual Learning Using Online Variational Bayes With Fixed-Point Updates). We will first reproduce these numbers on the deterministic-duration stream to confirm a faithful implementation, then run the heavy-tailed streams.
Strongest plasticity-preservation methods. L2-ER and continual backpropagation. They address long-stream rigidity by other means, and no scalar Permuted MNIST or Continual ImageNet accuracy for either is reported in the file. We will therefore run them ourselves.
Ablations isolating our two ingredients. One ablation removes the per-context trace normalization while keeping the fixed budget. The other removes the fixed budget while keeping the normalization.
Fairness of comparison. All methods use the same architectures and the same total step count. Each gets an equal hyperparameter search of 30 configurations, selected on the deterministic stream only and then frozen. This prevents any method from being tuned to the heavy tails.
What counts as winning. The balanced learner must have a sensitivity index statistically indistinguishable from one. It must also beat the best competitor's Pareto(1.5) average accuracy by more than seed noise. Finally, it must lose no more than one point on the deterministic stream.
Why not just…?
Why not just use online EWC with a normalized Fisher per task? This is the most serious threat. In the task-aware setting, normalized online EWC with decay is close to our normalization ingredient expressed as a penalty rather than a learning rate. Uncertainty-guided Continual Learning with Bayesian Neural Networks reports EWC at 95.78% on Split MNIST, but that run did not normalize for duration. We will include normalized online EWC as a full baseline, not a strawman. If it matches us when task identity is given, we will say so and treat the contribution there as the diagnosis plus the budget ingredient. The remaining distinct claim is the task-agnostic regime, where EWC requires task boundaries (Task-Agnostic Continual Learning Using Online Variational Bayes With Fixed-Point Updates).
Why not just put a floor under each weight's variance? A floor prevents total freezing, but it does not equalize contexts. A long task still owns most of the precision above the floor, so short-task forgetting should still track duration. The budget-without-normalization ablation is close to this, and we will also run a tuned floor. If either removes the sensitivity, normalization is unnecessary and the idea reduces to a simpler rule.
Why not just forget precision exponentially over time? Exponential discounting makes consolidation depend on recency weighted by duration. A short task followed by a long one is erased almost entirely, so this should be more duration-sensitive, not less. The prediction is directly testable on Pareto streams, and we include a tuned forgetting factor as a baseline.
Why not just replay old data? Replay sidesteps the problem at a storage and privacy cost. Reservoir replay is beaten by the buffer-free CONGA in continual test-time adaptation by only 0.6% (CONGA:Confidence-and-Gradient-Aware Learning Rate Schedule for Test Time Adaptation), so it remains a strong reference. We report a small reservoir buffer as an upper reference, not a competitor, and we test whether replay itself is duration-sensitive: reservoir sampling stores examples in proportion to duration.
Proposed Experiments
The decisive experiments run on three benchmarks, each with variable-duration streams generated by the protocol above:
- Permuted MNIST: 50 contexts.
- Split MNIST: contexts recur, so long revisits of one class pair can be tested.
- Continual ImageNet: binary tasks at the length scale where Loss of plasticity in deep continual learning reports standard backpropagation falling below a linear baseline after 2,000 tasks.
A fourth, exploratory benchmark is Slippery Ant with Pareto-distributed friction-regime lengths. Alongside accuracy metrics we track two signs of rigidity from Spectral Collapse Drives Loss of Plasticity in Deep Continual Learning: the effective rank of features and the fraction of near-zero Hessian eigenvalues.
The first decisive check is whether UCB and diagonal FOO-VB actually lose accuracy and plasticity as tails get heavier at fixed mean. If they do not, the premise fails and we report that as the finding.
The second check is whether the balanced rule flattens the sensitivity curve, and whether each ablation flattens only one of forgetting or rigidity, as the design predicts.
The third check is whether the linear-regression analysis predicts the measured slope against $\mathbb{E}[d^2]/\mathbb{E}[d]$.
The fourth check is how quickly task-agnostic insensitivity degrades as change-detector false alarms increase.
References
- Uncertainty-guided Continual Learning with Bayesian Neural Networks
- On performance bounds for balanced fairness
- Time-based Fairness Improves Performance in Multi-Rate WLANs
- Task-agnostic continual learning using online variational bayes with fixed-point updates
- Loss of plasticity in deep continual learning
- Spectral Collapse Drives Loss of Plasticity in Deep Continual Learning
- BAM: Bayes with Adaptive Memory
- MoCL: Metabolic Optimization for Curvature-Aware Continual Learning
- Confidence Based Dual Reinforcement Q-Routing: An adaptive online network routing algorithm
- CONGA:Confidence-and-Gradient-Aware Learning Rate Schedule for Test Time Adaptation