Keeping Gaussian Class Models Valid While LoRA Moves the Backbone

A classifier that stores one Gaussian (a mean and a covariance) per class can work surprisingly well on top of a pretrained network. It needs no stored old images, so it suits continual learning, where a model must keep adding classes without forgetting old ones. It works best if the features are first reshaped to look bell-shaped. This recipe has so far only worked on a frozen backbone, because the stored Gaussians stay correct only while the features do not move. The strongest recent continual learners do the opposite: they keep adapting the backbone with LoRA, a method that trains a small low-rank correction to each weight matrix and leaves the original weights untouched. In that setting the old classes' Gaussians go stale after every task.

This note proposes a way to have both. First, reshape the final features with a per-channel transform that is fitted once and then frozen, so the "Gaussian coordinate system" stays fixed. Second, after each task, update every old class's stored mean and covariance with a low-rank model of how the LoRA update moved features. That model is estimated only from current-task data. Everything happens at the final representation, so the LoRA weights still merge into the backbone at inference with no added cost.

Core Claim

In exemplar-free class-incremental learning with a LoRA-adapted Vision Transformer, a Gaussian class-conditional classifier will beat the prototype and linear heads used by current LoRA continual learners. This requires two conditions. The first is that the features are Gaussianized with a Yeo-Johnson transform (a per-channel power transform that, unlike square-root scaling, accepts negative values) whose parameters are frozen after the first task. The second is that old-class statistics are corrected with a low-rank affine drift map fitted on current-task features.

The contribution is not "Gaussianize something inside LoRA." It is a specific answer to an open problem stated by FeCAM: how to keep feature Gaussianization useful when the backbone drifts during fine-tuning.

Why this intersection, and why not the obvious one

Several papers motivate the gap. FeCAM combines Tukey's square-root transform with a Mahalanobis distance (a distance that accounts for each class's covariance) and covariance shrinkage on a frozen backbone, reaching 70.8% on CIFAR-100 with 10 tasks. FeNeC extends it to several clusters per class, again on a frozen backbone. ActiveDC uses the same power transform to calibrate class statistics for low-label fine-tuning, but never trains adapters. All three show that distribution shape matters for Gaussian classifiers. All three avoid backbone drift by construction.

A second group of papers supplies the adaptation machinery we reuse. LoRA⁻DRS subtracts past LoRA weights and projects gradients away from a subspace estimated by singular value decomposition (SVD) of features. This limits drift and reaches 77.94% average accuracy on 50-task ImageNet-R. E²-LoRA shows that the feature drift caused by LoRA is low-rank and concentrated, and uses this to order and prune ranks. Its drift observation is exactly what makes our drift map estimable from few samples.

The research file suggests an obvious combination: Gaussianize features before the SVD in LoRA⁻DRS or E²-LoRA. We do not pursue it, for two reasons. SVD minimizes squared reconstruction error for any distribution, so it does not rest on a Gaussian assumption. More importantly, forgetting is caused by $\Delta W x$ acting on the raw input $x$. A subspace found after a nonlinear per-channel transform does not correspond to a linear subspace of $x$, so projecting gradients against it no longer guarantees that $\Delta W x$ stays small on old data. Distribution shape matters where a density model is actually used, which is the classifier.

The Gaussianization layer of On the Importance of Gaussianizing Representations constrains the design in a different way. It puts Yeo-Johnson inside the network and reports gains over LayerNorm, for example 79.11% versus 73.25% on Food101. However, a nonlinearity placed between LoRA's two matrices would stop the adapter from merging into $W_0$. It would also add the training overhead that paper itself flags as a cost. A Unified Study of LoRA Variants warns that architectural additions to LoRA rarely survive a learning-rate sweep. We therefore leave LoRA's internals untouched.

Research Program

The backbone is a pretrained ViT-B/16 adapted with LoRA under the LoRA⁻DRS training procedure, which we use unchanged. Let $f_t(x)$ be the penultimate feature after task $t$. After the first task we fit per-channel Yeo-Johnson parameters by maximum likelihood on first-task features and then freeze them. This gives $z = g(f_t(x))$. Each class $c$ stores a mean $\mu_c$ and a shrunk covariance $\Sigma_c$ in $z$-space, and prediction uses the Mahalanobis rule of FeCAM.

At the end of task $t$ we still hold the previous LoRA weights, so each current-task image can be embedded both before ($z^{-}$) and after ($z^{+}$) the update. We fit a low-rank affine drift model by least squares:

$$z^{+} \approx z^{-} + U(M z^{-} + b),$$

Here $U \in \mathbb{R}^{d \times k}$ holds the top-$k$ left singular vectors of the drift matrix $Z^{+}-Z^{-}$. The matrix $M \in \mathbb{R}^{k \times d}$ and vector $b \in \mathbb{R}^{k}$ are fitted on current-task samples, and $k \ll d$. Writing $J = I + UM$, every old class is then updated as $\mu_c \leftarrow J\mu_c + Ub$ and $\Sigma_c \leftarrow J\Sigma_c J^\top$.

The low-rank restriction is what makes the fit feasible. A full $d \times d$ map would overfit the few thousand samples in a task. E²-LoRA's finding that drift concentrates in a few directions is the evidence that a small $k$ captures most of it.

The analysis has three parts. First, we measure per-channel normality ($R^2$ against a normal Q-Q line, the diagnostic used in the Gaussianization paper) across tasks, to test whether a Yeo-Johnson transform frozen after task one stays adequate as LoRA drifts. Second, we measure how much drift energy the top-$k$ directions capture. Third, we measure how far corrected old-class means sit from the true old-class means. The true means are computable offline for evaluation only, using held-out old data that the method never sees.

Baselines

The naive reference point is LoRA⁻DRS exactly as published, with its own prototype classifier and augmented triplet loss. It is run with the authors' schedule, and its learning rate is swept over the same grid we give our method. Because our method changes only the classifier and statistic updates, this comparison isolates the classifier.

The strongest existing competitor is E²-LoRA. It reports a Last-Acc of 78.58% on 50-task ImageNet-R (E²-LoRA), and its reported 82.77% on the shorter split matches joint training. We rerun it with the same backbone, splits, seeds, and learning-rate sweep. Beating it means a higher Last-Acc on 50-task ImageNet-R, averaged over at least three class orders, by more than the seed standard deviation. On the shorter ImageNet-R split, where E²-LoRA already matches joint training, the bar is to match it while keeping LoRA mergeable.

On CIFAR-100, the file reports backward transfer (BWT, the average accuracy change on old tasks) of −3.38 for LoRA⁻DRS versus −13.08 for plain LoRA (LoRA⁻DRS). No reported result was found on Lacuna for LoRA⁻DRS or E²-LoRA accuracy on CIFAR-100, so we will produce those numbers ourselves.

The frozen-backbone reference is FeCAM applied to the same pretrained ViT with no LoRA at all. It shows whether adaptation is worth its drift.

Three ablations isolate the key ingredients. The first keeps LoRA⁻DRS and the Gaussian classifier but applies no drift correction, which tests whether drift resistance alone keeps stored statistics valid. The second refits Yeo-Johnson at every task instead of freezing it. The third replaces Yeo-Johnson with the $\ell_2$ normalization of Mahalanobis++.

Why not just…?

Why not just run FeCAM on a frozen pretrained ViT and skip LoRA? This is a real threat on CIFAR-100, where pretrained features may already separate the classes. It is less likely on ImageNet-R, whose renditions sit far from the pretraining distribution, and where the LoRA methods report 77.94–78.58% on 50 tasks. The frozen-FeCAM baseline settles the question per dataset. If it matches us everywhere, the conclusion is that adaptation is not worth its drift, and we will report that.

Why not just rely on LoRA⁻DRS's drift resistance and store statistics once? If gradient projection truly kept old features in place, the drift correction would be unnecessary. The ablation without drift correction tests this directly. We also measure the distance between stored and true old-class means per task. A small, flat distance would make the drift map redundant. We expect the distance to grow with task count, because projection is based on intermediate-feature subspaces and only approximately constrains the final features.

Why not just $\ell_2$-normalize features, as Mahalanobis++ does? It is cheaper and fixes the dependence of Mahalanobis scores on feature norm. It does not remove per-channel skew, and FeCAM and ActiveDC both found power transforms necessary for covariance-based classifiers. The $\ell_2$ ablation, and a combined $\ell_2$-plus-Yeo-Johnson variant, decide the question. If $\ell_2$ alone matches, we adopt it and drop Yeo-Johnson.

Why not just sweep LoRA's learning rate harder, per the unified LoRA study? Our change leaves the adapter untouched, so any tuning gain in the baseline is also available to us. We give every method the same sweep, and we report accuracy as a function of learning rate, to show whether the classifier gain holds across the whole band rather than at a single lucky setting.

Proposed Experiments

Benchmarks are ImageNet-R with 10, 20, and 50 tasks, CIFAR-100 with 10 tasks, and CUB-200 and Cars-196, all exemplar-free. Metrics are Last-Acc, average incremental accuracy, and BWT, together with normality $R^2$ and drift-capture ratio as diagnostics. We also report training time added over LoRA⁻DRS, and confirm that inference cost is identical after merging.

The decisive check is 50-task ImageNet-R. The full method must beat E²-LoRA's Last-Acc and the no-correction ablation, and its advantage over the no-correction ablation should widen with task count. If the corrected and uncorrected versions tie, the drift map adds nothing and the idea fails. If frozen FeCAM ties the full method, the conclusion is that LoRA adaptation is unnecessary for Gaussian classifiers on that data.

A second check sweeps the drift rank $k$. Accuracy should saturate at small $k$ if E²-LoRA's low-rank drift observation carries over to final features. If $k$ must approach $d$, the premise that drift is low-rank fails for this purpose.