Order Sensitivity Is Approximation Error: Re-anchoring Bayesian Continual Learning

Exact Bayesian updating does not care about order. If a model sees task A and then task B, the true posterior is the prior times the likelihood of A times the likelihood of B, and multiplication commutes. Continual learning methods that call themselves Bayesian nonetheless show large swings in final accuracy when the same tasks arrive in a different order. Variational Continual Learning, for instance, concedes that it degrades badly under adversarial orderings. Continual learning here means training one model on a sequence of tasks without revisiting all old data. This note argues that such order sensitivity is not a property of the learning problem. It measures how much error the approximation has accumulated, and it can be reduced by fixing where the approximation is computed, not by choosing a better order.

The concrete proposal is to store each past task as a compact local quadratic summary of its loss: a gradient, a curvature estimate, and a tiny set of retained examples. Periodically, every stored summary is re-expanded around a single shared reference point, which we call the anchor. Sums of quadratics expanded at a common point are order-invariant by construction. The order dependence that remains comes only from how much each task's curvature changes between where it was learned and where it is re-expanded. That residual is a Taylor remainder, so it can be predicted and measured. The work therefore offers a mechanism that should reduce order sensitivity. It also offers a diagnostic: order disparity, read as a gauge of approximation error, can be checked against a ground-truth posterior.

Core Claim

In Gaussian-family recursive Bayesian continual learning, the dominant source of order dependence is expansion-point mismatch. Each task's likelihood is approximated around whatever parameters the model happened to hold when that task arrived. Different orders put those expansion points in different places. The exact posterior after $T$ tasks with datasets $\mathcal{D}_1,\dots,\mathcal{D}_T$ is

$$p(\theta \mid \mathcal{D}_{1:T}) \propto p(\theta)\prod_{t=1}^{T} p(\mathcal{D}_t \mid \theta),$$

which is identical under any permutation of $t$. Here $\theta$ are the network weights and $p(\theta)$ is the initial prior. Methods like Variational Continual Learning replace this product with a chain of projections. Each new posterior is fit with the previous posterior as its prior, so every projection error is baked in at a location that depends on history.

We instead keep task-wise factors and rebuild the aggregate around a shared anchor $m$, usually the current posterior mean:

$$\log q(\theta) = \log p(\theta) + \sum_{t=1}^{T}\Big[g_t(m)^\top(\theta - m) - \tfrac12 (\theta-m)^\top H_t (\theta - m)\Big].$$

In this expression $g_t(m)$ is task $t$'s log-likelihood gradient re-evaluated at the anchor on its small retained set. $H_t$ is a stored curvature summary of task $t$, either diagonal or Kronecker-factored. The sum has no memory of arrival order. The approximation departs from exact Bayes through two terms. One is the third-order Taylor remainder of each task's log-likelihood, which grows roughly as the cube of the distance between where $H_t$ was computed and $m$, since a second-order expansion is exact up to cubic terms. The other is noise in $g_t(m)$ from the small retained set. This yields a testable prediction. Order disparity in the standard sequential method should grow with how far parameters drift between tasks. Re-anchoring should remove the part caused by drift and leave a cubic residual. This also gives a Bayesian reading of the Max-Path and Periphery-to-Core rules from Optimal Task Order for Continual Learning of Multiple Tasks. Orders that alternate dissimilar tasks maximize drift and should hurt sequential projection most. Orders that finish on core tasks place the final expansion point near the consensus.

Research Program

The method borrows two ingredients and joins them in a new setting. The first is the idea of a common expansion point. Scalable Bayesian bi-level variable selection in generalized linear models builds an approximate Laplace approximation by Taylor-expanding every sub-model around one shared point, so that millions of marginal likelihoods cost almost nothing. That work operates on static data and has no notion of sequence. We use the same move to make task summaries commute. The second is retroactive correction of past-task parameters, which Scalable and Order-robust Continual Learning with Additive Parameter Decomposition and ORACLE use to cancel drift in a shared parameter block. Their correction is deterministic and architectural, with hard masks and no posterior. Ours is probabilistic, keeps a single shared network, and produces an approximate posterior with predictive uncertainty.

The motivation comes from three places. VCL's own admission about adversarial orders points to the gap. So does the 7× best-versus-worst order gap in the linear teacher-student analysis of Optimal Task Order. So does the observation in A PAC-Bayesian bound for Lifelong Learning that existing lifelong bounds assume tasks are drawn independently at random and say nothing about permutations. One constraint shapes the design. Online Curvature-Aware Replay uses Kronecker-factored curvature to smooth task transitions and reports that the factors become unstable at abrupt boundaries, needing resets and dampening. We therefore compute each $H_t$ once, at the end of its own task. We never update it through a boundary, and we only re-evaluate the cheap gradient term when re-anchoring.

The build has three parts. The first is the re-anchored learner itself. After each task it stores $H_t$ and a small retained set, around the 40 examples per task that VCL uses for its retained-example sets (which VCL calls coresets). Every $k$ tasks it recomputes all $g_t(m)$ at the current mean and refits the posterior to the summed quadratic plus the current task's exact loss. The second is an analysis tool. It measures, for any learner, the swap gap, defined as the difference in final posterior or accuracy when two adjacent tasks are exchanged. It then regresses this gap on measured inter-task parameter drift to test the cubic prediction. The third is a ground-truth setting where the true order-invariant posterior is available, so that order disparity can be compared directly with posterior error rather than only with accuracy.

Baselines

The naive reference point is plain sequential fine-tuning with no protection, run over the same set of task orders. It sets the scale of order disparity. The direct parent is Variational Continual Learning with its K-center retained set. It reports 93% on 10-task Permuted MNIST with the K-center coreset and 97% on Split MNIST. Bayesian Structural Adaptation for Continual Learning reports VCL at 94.87% on Permuted MNIST, so we will rerun VCL ourselves rather than trust either figure. Beating VCL means lower order disparity at equal or better mean accuracy, with the same retained-set size and network.

The strongest existing order-robust method is Additive Parameter Decomposition. On Split CIFAR-100, Scalable and Order-robust Continual Learning with Additive Parameter Decomposition reports about 60.7% accuracy and an Average Order-normalized Performance Disparity (AOPD), the per-task accuracy spread across orders, of 1.79%. EWC on the same benchmark reaches 7.06% AOPD. We will rerun it on identical splits and orders. We will give our method and APD the same tuning budget in trials, and report memory in bytes, since APD grows task-specific parameters while we store curvature factors and retained examples. Beating APD means matching its AOPD while also producing calibrated uncertainty. It also means matching it in a single shared network, at comparable memory, without per-task masks. Matching AOPD at clearly lower memory would also count. Approaching APD's disparity at modestly lower accuracy would be reported as a partial result, not a win.

The ablation that isolates the key ingredient is online Laplace with the same stored curvature but no re-anchoring. Each task's quadratic stays expanded at its original point, as in online EWC-style accumulation. If re-anchoring is what matters, this ablation should show the drift-dependent swap gap and our method should not. A second ablation re-anchors the gradient term but replaces the stored $H_t$ with curvature recomputed from the retained set alone. This tests whether the original-task curvature is worth storing.

Why not just…?

Why not just replay the retained examples instead of re-expanding around them? This is the most serious threat. The retained set does most of the work in both approaches, and VCL already shows that adding a K-center retained set lifts Permuted MNIST from about 90% to 93%. The proposal differs in what it does with the retained examples. It uses them only to re-evaluate a gradient at the anchor, while the curvature comes from the full original task. Plain replay instead treats 40 examples as a miniature training set. We will run replay, and VCL with replay, using exactly the same retained examples and memory. If replay matches us on AOPD and posterior error in the ground-truth setting, the claim that expansion-point mismatch is the bottleneck is wrong, and we will say so.

Why not just choose a good order upfront? HCTOS from Sequence Transferability and Task Order Selection in Continual Learning improves Split CIFAR-100 from about 0.74 to 0.77 average accuracy over random orders. The Max-Path and Periphery-to-Core rules can be estimated from 1% of each task's data. Both need every task in hand before training starts, which streaming settings do not allow. We include HCTOS-chosen and rule-chosen orders as conditions. The test is whether our method run on random or adversarial orders reaches the accuracy that the baselines only achieve on their best orders. If it falls short, upfront ordering remains the better tool whenever the task set is known.

Why not just freeze a backbone and add one adapter per task? Lifelong Language Learning with Adapter based Transformers removes order bias this way with under 15% parameter growth. Order invariance comes cheaply there because nothing shared is learned. That paper itself notes a remaining gap to joint multi-task training, and Learn Continually, Generalize Rapidly documents a capacity gap for frozen models. We will include a frozen-backbone adapter baseline on Split CIFAR-100 and measure backward transfer. Our method is only worth its cost if it shows positive backward transfer, meaning old tasks improve after new ones are learned, at disparity close to the adapter's.

Why not just use a better curvature-aware optimizer such as OCAR? OCAR reports 48.2% average anytime accuracy against 42.5% for the prior best method on 20-task Split-CIFAR100. Average anytime accuracy is accuracy averaged over all points during training, not only at the end. OCAR is a point-estimate method, and its authors note instability at task boundaries. We will run it on the same orders. Our prediction is that its order disparity tracks drift the same way the no-re-anchoring ablation does.

Proposed Experiments

The first experiment checks the mechanism against ground truth. We take the HIGGS data used by Hamiltonian Monte Carlo with Energy Conserving Subsampling, split it into sequential tasks by binning on selected features, and fit Bayesian logistic regression. That paper reports HMC-ECS running 478.7 to 554.1 times faster than full-data HMC with indistinguishable posteriors. This makes it practical to compute the exact, order-invariant joint posterior as a reference. We also include a small linear-Gaussian teacher-student model where every method should be exact, as a sanity check. Across all task permutations for small $T$, and sampled permutations for larger $T$, we measure KL divergence to the reference posterior and the swap gap. We then test whether the gap scales with the cube of inter-task drift for re-anchored factors and with a lower power for sequential projection.

The second experiment tests whether the mechanism holds in deep networks. We use Permuted MNIST and Split MNIST against VCL's reported numbers. We use Split mutual-CIFAR-10, where adjacent tasks share a class. We use 10-task Split CIFAR-100, where the APD and EWC disparity figures give external anchors. For each benchmark we evaluate random orders, a Max-Path adversarial order, a Periphery-to-Core favorable order, and an HCTOS order. We report mean accuracy, AOPD, backward transfer, expected calibration error, and memory.

The decisive checks are three. The re-anchored learner must cut AOPD substantially below VCL and the no-re-anchoring ablation on Split CIFAR-100, and approach APD's reported 1.79%. Replay with identical memory must not close that gap. In the HIGGS setting, order disparity must track measured posterior error across methods. If disparity and posterior error come apart, the central reinterpretation of order sensitivity fails.