Routing-Free Channel Recalibration for Few-Shot Class-Incremental Learning

Many continual learning systems give each task its own small set of extra parameters so that learning a new task cannot overwrite an old one. One cheap version rescales the feature channels of a frozen network, turning up the channels that matter for the task and turning down the rest. The price is that at test time someone must say which task an image belongs to, so the system knows which set of channel weights to apply. This is the "task oracle." Remove the oracle and the system has to guess the task. When new classes arrive with only one to five labeled images each, which is the regime of few-shot class-incremental learning (FSCIL), there is too little data to learn a reliable guesser, and a wrong guess sends the image through the wrong channel weights.

We propose to remove the guess instead of improving it. The channel weights are computed from two things available at test time with no task label: the query image's own features, and the running bank of class prototypes, where a prototype is the mean feature vector of a class. We apply the rescaling at the final embedding. A per-channel scale commutes with averaging, so the scaled prototype of any old class equals the scale applied to its stored mean. Old classes can therefore be re-weighted exactly under every new set of channel weights, with no stored images, no per-task modules, and nothing to route between. The central empirical question is whether this conditioned recalibration beats a strong frozen-feature nearest-prototype classifier and a fixed per-channel metric. If it does not, that negative result is useful in its own right.

Core Claim

In rehearsal-free FSCIL, channel recalibration should depend on the query and on the set of classes seen so far, not on a task identity. Concretely, a small squeeze-and-excitation-style network $g$ (a two-layer bottleneck network that outputs one weight per channel) is meta-trained on base classes to produce weights $s = g(z_q, B) \in \mathbb{R}^C_{>0}$. Here $z_q$ is the query's $C$-dimensional embedding from a frozen backbone, and $B$ summarizes the prototype bank. That summary contains the per-channel spread of prototypes across classes, the per-channel within-class spread estimated on base data, and each class's shot count. Classification then uses

$$ \hat y = \arg\max_c \; \cos\big(s \odot z_q,\; s \odot \mu_c\big), $$

where $\mu_c$ is the stored unscaled mean embedding of class $c$ and $\odot$ is elementwise product. Since $\frac{1}{K}\sum_k s\odot z_k = s\odot\mu_c$, re-weighting old classes is exact. Adding a session adds $C$ numbers per new class and zero parameters.

The claim is new relative to the closest work in the file. Task-Oriented Channel Attention for Fine-Grained Few-Shot Classification (TDM) computes channel weights from support-set class discrepancies inside isolated few-shot episodes. It shows that such weights help on fine-grained data: on Aircraft with a Conv-4 backbone, ProtoNet rises from 49.88% to 52.88% at 1-shot. TDM never faces a growing class set, forgetting, or task inference. Our proposal reuses its insight as the method source and moves it into the sequential setting, where the "support set" becomes the persistent prototype bank. The commutation property is what makes that move exact for old classes.

Relation to Prior Work

Three lines of work motivate the proposal by showing where the oracle bites.

CLR: Channel-wise Lightweight Reprogramming for Continual Learning inserts per-task depthwise channel kernels behind a frozen ResNet. It reaches 86.05% average accuracy on a 53-dataset sequence, but it assumes the oracle and full-size datasets. Conditional Channel Gated Networks for Task-Aware Continual Learning removes the oracle with a task classifier over feature maps, but that classifier needs replayed exemplars and its gates are binary. On the Effectiveness of LayerNorm Tuning for Continual Learning in Vision Transformers tunes per-task LayerNorm scales and shifts, which is itself a form of channel recalibration, and selects them with learned keys. It observes a "neighborhood effect" that softens misrouting. All three keep one parameter set per task and must select among them; we keep none.

Two other lines constrain the design. Supermasks in Superposition infers the task by minimizing output entropy over a mix of per-task masks. That works when wrong modules give uncertain outputs. The file flags that few-shot classifiers tend to be overconfident, so entropy routing is a design we expect to fail here, and we test it as a baseline. Few-Shot Class Incremental Learning with Attention-Aware Self-adaptive Prompt (ASP) reports that prompt methods with learned selection keys, such as L2P and CODA-Prompt, score near 0% harmonic accuracy on few-shot CIFAR-100. This is direct evidence that per-task selection mechanisms break when each task has five examples, and it is the main reason we avoid selection entirely.

The FSCIL methods we compete with are the following. Few-shot Continual Infomax Learning uses mutual-information objectives with prototype classification in a single fixed embedding. MASIL fixes classifier geometry to a maximally separated frame and builds new classes from a factorized concept bank. A Bag of Tricks for Few-Shot Class-Incremental Learning combines contrastive pre-training, reserved prototype slots, and static subnetwork tuning. None of them changes the channel weighting per query at inference. Our method is complementary: it can sit on top of any of their embeddings.

Research Program

The work has three parts.

The first part builds the recalibration network and its training. On base classes we run pseudo-incremental episodes. Each episode draws a random subset of base classes as "old," whose prototypes come from all their images. It draws a disjoint subset as "new," whose prototypes come from $K\in\{1,5\}$ shots. Queries come from both groups and are classified over the union of classes. This matches test conditions, including noisy few-shot prototypes. Passing each class's shot count into the bank summary lets $g$ learn to trust five-shot channels less than full-data ones, which addresses the unreliable-statistics problem the file raises for small samples. The backbone is frozen after base training.

The second part is analysis of what the weights do. We will check two things: whether $s$ changes smoothly as the bank grows, and whether adding a session ever lowers old-class accuracy. Because the backbone and old prototypes are fixed, any old-class drop is caused by $s$ alone, so it can be measured cleanly.

The third part is a controlled comparison against the routing alternatives. We implement per-session channel-scale modules and select among them by oracle, by nearest-prototype task inference, and by entropy minimization. This measures the "oracle gap" and the misrouting rate as sessions and shots vary. It turns Gap 3 of the file, the untested tension between entropy routing and few-shot overconfidence, into a measured result.

Baselines

The naive reference point is the same frozen backbone with cosine nearest-class-mean over all seen classes, with no recalibration. Every other method must beat it. On strong pre-trained backbones it is hard to beat, so it is the real bar and not a straw man.

The ablation that isolates the key ingredient is a fixed diagonal metric. Per-channel weights are computed once from base-class statistics, using the inverse within-class spread, and never change. We also run two half-conditioned variants: a query-only network, which is plain squeeze-and-excitation on the query, and a bank-only network, which uses no query input. If the full model does not beat the fixed metric by a clear margin, the conditioning idea fails. We set that margin in advance at 1 point of final-session accuracy averaged over five seeds, outside seed variance.

The routing family consists of per-session channel-scale modules trained on each session's shots and selected three ways. The oracle version is an upper bound that is unavailable in practice. Prototype-distance task inference is the HiDe-Prompt-style inference step, adapted from Hierarchical Decomposition of Prompt-Based Continual Learning. SupSup-style entropy minimization is the third. Beating the two practical routers, and closing most of the gap to the oracle, is the evidence that removing routing is better than improving it.

The strongest existing methods depend on the backbone.

  • ResNet-18 trained from scratch on base classes, miniImageNet. Few-shot Continual Infomax reports 52.76% final-session accuracy and a knowledge-retention score of 69.11%. MASIL reports only a relative +3.16% over ALICE, so no absolute number was found on Lacuna for MASIL.
  • ResNet-18, CIFAR-100 and CUB-200. No absolute FSCIL final-session number was found on Lacuna for these methods. We will rerun them from public protocols.
  • Frozen ViT. ASP reports 86.7% final accuracy, 85.3% harmonic accuracy, and 5.5% performance drop on CIFAR-100.

Fairness rules apply to all comparisons. The same backbone and base-training recipe feed every prototype-based method. Recalibration methods, including ours, get an equal hyperparameter search budget of 30 configurations on a held-out split of base classes. When we cite a paper's number, we also report our reproduction under our pipeline. Beating these methods means higher final-session accuracy and harmonic accuracy, the average of base-class and novel-class accuracy, over five seeds on the same splits.

Why not just…?

Why not just use nearest-class-mean on a frozen strong backbone? This is the most serious threat. ASP's 86.7% on CIFAR-100 with a frozen ViT suggests strong features already do much of the work. The one piece of evidence in the file that channel weighting still adds value comes from TDM: +3 points at 1-shot on Aircraft over ProtoNet, in the fine-grained regime. We therefore expect gains to concentrate on CUB-200 and at 1-shot, and to be small on CIFAR-100 with a ViT. If our method does not beat nearest-class-mean on CUB, we will report that channel recalibration is unnecessary with strong frozen features.

Why not just learn a fixed per-channel metric, such as variance normalization? Our method only helps if the useful channel weights change with the query and with which classes are present. Otherwise a fixed metric captures everything at lower cost. The fixed-metric ablation settles this directly under the pre-set margin. We also test the hypothesis behind conditioning with a second analysis. If conditioning matters, $s$ should shift toward fine-grained channels when a new session adds classes similar to existing ones. We can measure this by correlating the change in $s$ with the minimum distance from the new class to its nearest old class.

Why not just keep per-session channel modules and route with prototype distances? Prototype-distance routing is cheap and rehearsal-free, and the LayerNorm-tuning neighborhood effect suggests misrouting may cost less than feared. The counter-evidence is ASP's report of near-0% harmonic accuracy for key-based prompt selection under few shots. That result is about prompt keys, not prototype routing, so it is suggestive only. Our routing-family experiment measures misrouting rate and accuracy against the oracle directly. If prototype routing with per-session modules matches our method, the conclusion becomes that routing is fine and our contribution reduces to a cheaper equivalent.

Why not just fine-tune one shared recalibration layer each session with a small learning rate? That would reintroduce forgetting of old classes, since the layer's weights drift with each session. It would also fit five images per class. We include it as a baseline to show how much old-class accuracy it loses per session relative to the exact re-weighting our design allows.

Proposed Experiments

We use the standard FSCIL splits.

  • miniImageNet and CIFAR-100: 60 base classes, then eight 5-way 5-shot sessions.
  • CUB-200: 100 base classes, then ten 10-way 5-shot sessions.
  • 1-shot variants of all three.

Each split runs on two backbones: a ResNet-18 trained from scratch on base classes, and a frozen pre-trained ViT. Metrics are per-session accuracy, final-session accuracy, harmonic accuracy, and performance drop, with five seeds.

Four checks decide the outcome.

  1. Old-class effect. Old-class accuracy is recorded after each session. Our method should never lose more than 0.5 points on old classes from a session's addition, because only $s$ can change.
  2. Oracle gap and misrouting. Misrouting rate and the gap to the oracle are plotted against number of sessions and shots for the routing baselines.
  3. Entropy-routing overconfidence. For each routed query we record whether a wrong module yields lower entropy than the right one. This directly tests whether few-shot overconfidence breaks entropy routing.
  4. Where gains land. Improvements over nearest-class-mean should be largest on CUB at 1-shot. If they are uniform across datasets or absent on CUB, the explanation that conditioned recalibration isolates fine-grained cues is wrong, and we will say so.