When a classifier learns new classes one batch at a time, it tends to overwrite what it knew. Recent theory shows that this forgetting is not spread evenly. It concentrates in a handful of directions in the model's output space, and penalizing change only along those directions preserves old tasks far better than generic weight-space penalties. Those directions are found by ranking the eigenvectors of a similarity matrix built from the training data by how much energy they carry. Real data is long-tailed: a few "head" classes have thousands of images and many "tail" classes have a handful. In that setting, energy-ranked directions describe the head classes almost exclusively. Our hypothesis is that the standard way of finding "where forgetting lives" is blind to rare classes. Over 30 to 50 learning steps, rare classes learned early would then be erased while the regularizer reports that everything important is protected.
We propose to test this hypothesis directly and to repair it with a small change. The repair is to choose the protected directions after reweighting the similarity matrix so that every old class counts equally, rather than every old example. If the hypothesis holds, this class-balanced selection should keep tail accuracy alive over long horizons at the same plasticity cost as the unweighted version. If it fails, the measurement itself is a useful negative result: it would show that frequency-aware calibration, not protection of the representation, is the real bottleneck.
Core Claim
Catastrophic Forgetting is Low-Rank analyzes continual learning through the neural tangent kernel, the matrix of gradient similarities between examples that governs how a wide network's outputs move during training. The paper shows three things:
- Forgetting energy sits in 1 to 6 eigenmodes.
- A closed-form predictor built from the cross-task kernel matches actual forgetting with cosine similarity above 0.99 on frozen-backbone models.
- Penalizing drift along the top-$k$ eigenvectors of the previous task's kernel reaches about 78% on shared-head Split-MNIST, where EWC and SI reach about 20%.
That analysis assumes balanced classes. This paper supplies our method and is also the paper we stress-test.
Our claim is specific. Under long-tailed incremental streams, the share of tail-class forgetting energy that falls outside the top-$k$ energy-ranked modes grows with the imbalance ratio and with the number of tasks. Selecting modes from a class-reweighted kernel recaptures that share, with no increase in the number of protected directions.
Concretely, let $K$ be the kernel (or, for a frozen backbone with trained adapters, the Gram matrix of adapter Jacobian features) over stored examples of old classes. Let $W$ be a diagonal matrix assigning each stored example the weight $1/n_c$, where $n_c$ is the number of stored examples of its class $c$. We protect the top-$k$ eigenvectors $u_1,\dots,u_k$ of $W^{1/2} K W^{1/2}$ instead of $K$, and train each new task with
$$ \mathcal{L} = \mathcal{L}_{\text{new}} + \lambda \sum_{i=1}^{k} \left( u_i^\top W^{1/2} \left[ f_\theta(X_{\text{old}}) - f_{\text{old}}(X_{\text{old}}) \right] \right)^2 . $$
Here $f_\theta$ is the current model, $f_{\text{old}}$ is a frozen snapshot of the model's outputs on the stored inputs $X_{\text{old}}$, and $\lambda$ is the penalty weight. The snapshot is taken when each class was last trained, not from the immediately preceding checkpoint. This avoids the step-to-step "telephone game" of chained distillation noted by Xia et al. and Klingner et al.. The only differences from the original spectral regularizer are $W$ and the anchoring of the snapshot. The ablations below isolate both.
Relation to Existing Long-Tailed Incremental Work
Three papers already combine long-tailed data with class-incremental learning, and they motivate rather than anticipate this proposal.
- Long-Tailed Class Incremental Learning defines the Ordered and Shuffled protocols. It observes that even "balanced" incremental learning is long-tailed, because a small replay buffer faces a full stream of new data. It fixes classifier bias with a second training stage that learns per-class weight scaling.
- DAT argues that from-scratch representations cannot be repaired post hoc. It uses a frozen ViT with dynamic adapters and a balanced/imbalanced dual-path distillation.
- DELTA handles the single-pass online version with contrastive representations and an equalization loss.
All three correct the classifier or the training signal. None asks which directions of the representation are protected against forgetting, and none goes beyond 25 tasks.
CLTR runs the arrow the other way. It uses continual learning to solve offline long-tail recognition, and proves that joint training leaves weights within $O(1/\sqrt{IF})$ of the head-only solution, where $IF$ is the imbalance factor. We read that bound as further evidence for our concern: any quantity estimated from imbalanced data, a kernel's eigenvectors included, will be dominated by head classes.
Long-horizon stability methods such as GR-LoRA reach 50 tasks only on balanced streams. GR-LoRA projects gradients orthogonally into a shared backbone and recycles the rest into per-task low-rank adapters. GR-LoRA has the same blind spot as energy-ranked spectral protection: its recycled gradient is dominated by head classes, which PaCo shows carry most of the gradient norm under imbalance.
Research Program
Stage one is measurement. It is the part most likely to change the field's practice whether or not the method wins.
On each long-tailed stream we train a strong adapter-based learner and log, at every task boundary, the actual output change on every old class. We decompose that change onto the eigenvectors of the unweighted kernel and report the fraction of forgetting energy captured by the top-$k$ modes, separately for head, medium and tail classes. We also report it as a function of how many tasks have passed since the class was learned. We verify the cross-task predictor's cosine accuracy per frequency band, since the 0.99 figure was established only on balanced data.
The hypothesis predicts that coverage of tail forgetting falls with imbalance ratio and horizon, while coverage of head forgetting stays high. The Kronecker rule $k^* \approx C \cdot k_G$ from the low-rank paper (vulnerable rank roughly equals class count times effective feature rank) gives a second prediction. Measured under the reweighted kernel, the vulnerable rank should rise toward that value, because tail classes contribute effective rank that the unweighted spectrum hides.
Stage two is the intervention described in the Core Claim, with three design points.
- Where the kernel is computed. For a frozen backbone with a trained classifier we use the exact predictor. With trained adapters we use a random projection of adapter-parameter Jacobians on the stored examples, and check approximation quality against measured forgetting.
- Weights for classes with no stored examples. Tail classes may have fewer stored examples than the buffer allows. We therefore set $W$ from stored counts, not stream counts, and test both choices.
- Cost. The eigen-decomposition is over the buffer only, so it is independent of task count. Unlike entropy routing over all historical modules, inference cost does not grow with the number of tasks.
Baselines
Naive reference. Sequential fine-tuning of the same pretrained ViT with adapters, with the same replay buffer and plain cross-entropy. This gives the floor for both forgetting and tail collapse.
Frozen-backbone balanced closed-form classifier. In the spirit of ACIL and DS-AL, which update linear weights analytically so that incremental training matches joint training on fixed features, we add per-class reweighting so tail and head classes count equally. This baseline cannot forget by construction, so it is the most dangerous competitor whenever the pretrained features already separate the tail. We run it on identical features and report it at every horizon.
Strongest existing long-tailed incremental method. DAT reports 71.70% average incremental accuracy on CIFAR-100, 10 tasks, imbalance ratio 0.01. We first reproduce that number to validate our implementation, then extend DAT to 20 and 50 tasks with its own hyperparameter search budget.
Two-stage calibration. We add the weight-scaling stage of Liu et al., which reports gains of 6.28 and 6.44 points over LUCIR and EEIL on Shuffled ImageNet-Subset. We add it on top of both our method and the naive learner, because calibration and representation protection are complementary rather than rival.
Strongest balanced long-horizon method. GR-LoRA reports 76.74% final accuracy on balanced 50-task ImageNet-R. We run it unchanged on our long-tailed version.
No reported result was found on Lacuna for any method on long-tailed 50-task ImageNet-R or 50-task CIFAR-100-LT, so all long-horizon numbers will be produced here under one protocol.
Key-ingredient ablation. The decisive ablation is unweighted spectral regularization from the low-rank paper, at the same $k$ and also at a larger $k$ chosen so its plasticity cost (accuracy on the newest task) matches ours. We also run class-balanced replay plus distillation from the immediately preceding checkpoint, which separates the effect of anchoring the snapshot from the effect of reweighting.
Fairness and the success criterion. All methods share the backbone, buffer size, augmentation and an equal number of hyperparameter trials, tuned on a held-out task ordering. Beating a baseline means a higher final tail-class accuracy and a higher balanced average incremental accuracy at 50 tasks, across three orderings, with no loss on head classes beyond the seed-to-seed spread.
Why not just…?
Why not just freeze a foundation model and fit a balanced linear classifier or class prototypes? This is a real threat, and in the frozen-feature regime it may simply win, since a closed-form balanced classifier on fixed features does not forget. DAT reports that frozen-backbone prompt methods still degrade under imbalance at ratio 0.01 when target domains are far from pretraining, which is why ImageNet-R renditions are our main benchmark. Our experiments report the frozen balanced classifier at every horizon. If it matches adapter-based learners on ImageNet-R-LT, the honest conclusion is that representation protection is unnecessary there. We would then restrict our claim to domains where adaptation clearly helps.
Why not just protect more eigenmodes? Raising $k$ in the unweighted regularizer will eventually cover tail directions, at the cost of freezing many head directions the new task needs. The ablation that matches plasticity cost answers this directly. If unweighted protection at matched cost equals ours, reweighting adds nothing and we would say so.
Why not just oversample the tail in the replay buffer and use a balanced loss like Seesaw? Rebalancing the loss changes which classes drive the update on the current task. It does not decide which old functions are held fixed. The buffer remains small relative to incoming data, as Liu et al. note, and DELTA finds that pairing more stored examples saturates beyond about ten per sample. Seesaw-weighted balanced replay with chained distillation is included as a baseline. The stage-one measurement shows whether its tail forgetting is concentrated in modes it fails to hold.
Proposed Experiments and Decisive Checks
Benchmarks.
- CIFAR-100-LT at imbalance ratios 0.1, 0.01 and 0.005, in 10, 25 and 50 tasks, under both Ordered and Shuffled protocols.
- A long-tailed ImageNet-R built by subsampling each of the 200 classes to a power-law profile at ratio 0.01, in 10, 25 and 50 tasks.
- An iNaturalist species subset for a natural rather than synthetic tail, addressing the concern of Sun et al. that step imbalance is unrealistic.
Metrics. Average incremental accuracy, final accuracy split into head, medium and tail classes, a forgetting curve indexed by time since a class was learned, and accuracy on the newest task as the plasticity measure.
First decisive check. Stage one should show a falling top-$k$ coverage of tail forgetting under the unweighted kernel as imbalance and horizon grow. If coverage stays high, the premise is false and the method should not be pursued.
Second decisive check. At 50 tasks and ratio 0.01, class-balanced selection should beat unweighted selection at matched plasticity on early-learned tail classes, and the gap should widen with horizon. Equal performance at 10 tasks but separation at 50 is the specific long-horizon signature we expect.
Third decisive check. Reweighted spectral protection followed by weight-scaling calibration should beat both components alone. This would show that protecting the representation and recalibrating the classifier fix different failures.
References
- Catastrophic Forgetting is Low-Rank: A Function-Space Theory for Continual Adaptation
- Learn and Review: Enhancing Continual Named Entity Recognition via Reviewing Synthetic Samples
- Class-Incremental Learning for Semantic Segmentation Re-Using Neither Old Data Nor Old Labels
- Long-Tailed Class Incremental Learning
- Dynamic Adapter Tuning for Long-Tailed Class-Incremental Learning
- DELTA: Decoupling Long-Tailed Online Continual Learning
- On The Relationship Between Continual Learning and Long-Tailed Recognition
- GR-LoRA: Gradient-Recycling Low-Rank Adaptation for Class-Incremental Learning
- Parametric Contrastive Learning
- ACIL: Analytic Class-Incremental Learning with Absolute Memorization and Privacy Protection
- DS-AL: A Dual-Stream Analytic Learning for Exemplar-Free Class-Incremental Learning
- Seesaw Loss for Long-Tailed Instance Segmentation
- Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspective