Recency-Free Federated Class-Incremental Learning from Encrypted, Fixed-Size Class Statistics

When a neural network learns new classes one batch at a time, it does not forget old ones evenly. Its final classification layer drifts toward whatever it saw most recently, so an image of an old class gets labeled as one of the newest classes. This failure is called task-recency bias. The best known fixes all need some view of class-level information: class counts, class means, or per-class softmax groupings. Now place this problem on many phones or hospitals that each see new classes over time and must pool what they learn without revealing their data. The privacy tools that make such pooling cheap are encryption schemes whose ciphertext size does not grow with the number of participants. They can only add numbers together, and their keys are fixed when the system is set up. Under those constraints the usual recency-bias fixes cannot run, and the set of classes keeps growing after the keys are fixed.

This note proposes a way through. Each client turns its data into a single fixed-size summary: a hashed sketch of per-class feature sums and counts, plus a pooled feature second-moment matrix. The summary has the same shape however many classes the client has seen and however many classes exist globally, and it combines across clients by plain addition. The server only ever decrypts the sum. From that sum it builds a nearest-class-mean or shared-covariance Gaussian classifier, which has no learned weight vectors whose norms can inflate for new classes. The claim is that this one additive statistic does two jobs. It removes recency bias at the federation level, including the case where clients holding old classes drop out. It also fits encryption whose ciphertext stays constant in the number of participants and whose setup never has to be redone as new classes appear.

Core Claim

Recency bias in class-incremental learning has two separable sources. The first is a parametric classifier head whose new-class weights receive dense positive gradients while old-class weights starve. The global softmax makes this worse, as SS-IL: Separated Softmax for Incremental Learning shows analytically. The second is feature collapse in the backbone, which skews covariance estimates, as Task-recency bias strikes back shows. In a federation there is a third source that, as far as this file shows, has not been studied. If the clients who hold old classes participate less over time, any aggregate of recent model updates is dominated by clients seeing new classes. We claim that replacing the parametric head with a classifier rebuilt from running, additively aggregated sufficient statistics removes the first and third sources. Adding a pooled second-moment term partially addresses the second. The whole construction requires only additive homomorphism over a payload whose size is fixed at setup.

Let $\phi(x)\in\mathbb{R}^d$ be the backbone feature of input $x$. Let $h:\text{labels}\to\{1,\dots,m\}$ be a hash into $m$ slots and $s:\text{labels}\to\{\pm1\}$ a sign hash. Each client $k$ uploads the following, all encrypted elementwise:

$$S_k=\sum_{i\in D_k} s(y_i)\,\phi(x_i)\,e_{h(y_i)}^{\top},\qquad n_k=\sum_{i\in D_k} s(y_i)\,e_{h(y_i)},\qquad M_k=\sum_{i\in D_k}\phi(x_i)\phi(x_i)^{\top}.$$

Here $D_k$ is the client's new data since its last upload, and $e_j$ is the $j$-th unit vector. The server decrypts only $\sum_k S_k$, $\sum_k n_k$ and $\sum_k M_k$, and keeps a running total across rounds. For any class $c$ it estimates the class mean as $\hat\mu_c=s(c)\,S_{:,h(c)}/(s(c)\,n_{h(c)})$. This is a count-sketch estimate: hash collisions add zero-mean noise, and that noise shrinks as $m$ grows, with several independent hash rows taking the median. The payload has $m(d+1)+d^2$ entries no matter how many classes exist, so the encryption parameters fixed at setup never need to change. This addresses the static-universe tension that constant-size attribute and broadcast schemes impose.

Relation to Prior Work

Several papers motivate the method. Supervised Contrastive Replay replaces the softmax head with a nearest-class-mean classifier and reports large gains in online class-incremental learning. That is evidence that the class-mean route avoids head-induced recency bias, but SCR computes means centrally from a replay buffer. AdaGauss shows that class means alone fail without exemplars, because covariances shift and feature dimensions collapse. This finding constrains our design and is the reason we aggregate a second-moment matrix rather than means only. SS-IL, the focal-loss method AFS in New Insights on Relieving Task-Recency Bias, and the calibration tricks in Rethinking Experience Replay all correct the parametric head. Each needs per-class counts or per-sample confidences during training, and in a federation those quantities are exactly what secure aggregation hides. We do not correct the head; we remove it. CLIB decouples training from the stream by training only on a balanced buffer. Our running statistics play the role of a balanced memory that contains no stored examples. The federated continual learning systems Asynchronous Federated Continual Learning and Federated Continual Novel Class Learning handle the same distributed, non-stationary setting. They communicate model updates rather than encryption-compatible fixed-size statistics, and they do not study dropout that correlates with task.

The cryptography is reused, not invented. Turning HATE Into LOVE gives an additively homomorphic threshold scheme whose ciphertext is constant in the number of participants and which tolerates participants vanishing mid-protocol. Its aggregation protocol takes 3 rounds against 5 for Google's Practical Secure Aggregation. The same paper also constrains us in two ways. Its fully sender-compact construction relies on indistinguishability obfuscation, which is impractical; its practical instantiations are additive-only, built on ElGamal or Paillier encryption. That is why every quantity we aggregate must be a sum. Its impossibility result also means recipient sets cannot be hidden at constant size, so we do not claim that participation is hidden from participants.

Research Program

We will build the client-side statistic and a server that rebuilds the classifier from it. The classifier is a nearest-class-mean rule plus a shared-covariance Gaussian rule. The shared covariance is $\Sigma=\overline{M}-\sum_c \pi_c\hat\mu_c\hat\mu_c^{\top}$, with shrinkage toward the identity to counter feature collapse, where $\overline{M}$ is the aggregated second moment normalized by the total count and $\pi_c$ is the estimated fraction of samples in class $c$. Features are encoded in fixed point for the additive scheme. We will measure the accuracy cost of quantization separately.

We will study two backbone regimes. In the first, the backbone is frozen and pretrained, so statistics never go stale and the question is purely about the classifier and the federation. In the second, the backbone is updated by federated averaging. Old statistics then go stale as features drift. We handle this by having clients periodically recompute statistics on whatever data they still hold. In a federation, unlike a single learner, old classes often survive on some clients. We will also test a covariance-adaptation correction in the spirit of AdaGauss for classes that no client still holds.

The analysis has two parts. The first bounds how hash collisions in the sketch affect mean-estimation error and therefore nearest-class-mean accuracy, as a function of $m$ and the number of classes. The second measures recency bias directly, as the fraction of old-class test inputs predicted as belonging to the most recent task. We record this under three dropout schedules: random dropout, dropout correlated with task (clients with old classes leave first), and no dropout.

Baselines

The naive reference point is federated averaging of a standard backbone with a linear softmax head, trained on each client's stream, with no recency correction and no encryption. This shows how much bias the federation induces by default.

The strongest competitor on the recency question is a set of centralized-quality corrections run in plaintext inside the federation. One is SS-IL's separated softmax applied locally on each client. Another is SCR's nearest-class-mean computed from plaintext pooled statistics, which is a privacy-violating oracle for our method. A third is AdaGauss run locally. As reference points from the file, SS-IL reports 64.5% top-1 on ImageNet-1K with 10 tasks against EEIL's 59.4%, and SCR reports 32.8% average accuracy on Split CIFAR-100 with a 2,000-sample buffer against ASER-NCM's 25.2%. AFS reports 37.6% on Mini-ImageNet with a 5,000-sample buffer. CLIB reports an $A_{AUC}$ of 46.67% on i-Blurry CIFAR-100, where $A_{AUC}$ is accuracy averaged over the whole stream as an area under the curve. All of these are centralized results. We found no reported result on Lacuna for any of these methods in a federated or encrypted setting, so we will rerun them ourselves. Every method gets the same client partitions, backbone, number of communication rounds, local compute, and hyperparameter search budget, with SS-IL and SCR given replay buffers of matched total size. Our method beats this group if it matches the plaintext nearest-class-mean oracle within one point of average accuracy, outperforms local SS-IL, AFS and AdaGauss when dropout is correlated with task, and does so without storing any exemplars.

On the systems side, the baseline is the same statistic aggregated with Google-style Practical Secure Aggregation, using SecAgg's reported 5-round structure. We implement this baseline ourselves and measure rounds, per-client bytes, and completion rate under dropout. The HATE-based version beats it if per-client upload stays flat as participants scale while SecAgg's cost grows, and if aggregation completes under dropout rates where SecAgg stalls.

The ablations isolate each ingredient. Exact per-class statistics stand against the sketch at several values of $m$, which prices the fixed-size design. Means only stand against means plus pooled covariance, which tests whether the AdaGauss collapse problem appears here. Running accumulation stands against per-round statistics, which tests whether accumulation is what neutralizes correlated dropout.

Why not just…?

Why not just do plaintext federated averaging and apply SS-IL or a bias-correction layer on each client? This is the most direct threat, and it may match us on accuracy when every client sees every class. It fails in two places. Locally corrected heads still get averaged with weights dominated by whichever clients are active, so dropout correlated with task should reintroduce recency bias at the server. A bias-correction layer would need the global class histogram, which is a privacy leak. The correlated-dropout experiment will show which of these effects is real.

Why not just use standard secure aggregation instead of constant-size threshold encryption? Our statistic works with both, and we say so plainly: the learning contribution does not depend on which encryption scheme is used. The case for HATE rests on round count and tolerance to vanishing participants. Turning HATE Into LOVE gives 3 rounds against 5 and dropout tolerance, but its fully compact form needs indistinguishability obfuscation. If our measurements show SecAgg costs about the same at realistic federation sizes, we will report that the encryption choice does not matter at those sizes.

Why not just reserve a fixed number of class slots and send exact per-class statistics? This works if an upper bound on the number of classes is known at setup. In that case the sketch helps only when the class universe outgrows that bound. The ablation of exact statistics against the sketch measures what the sketch costs, and we will recommend exact slots wherever a bound is available.

Why not just add differential-privacy noise to model updates? That protects individuals but does not change the parametric head that causes recency bias. It is complementary: noise can also be added to our sums, and we will report accuracy under matched privacy noise.

Proposed Experiments

We will split Split CIFAR-100, i-Blurry CIFAR-100, TinyImageNet, and ImageNet-Subset across 20 to 500 simulated clients, with heterogeneous class schedules. The metrics are average accuracy, $A_{AUC}$, the old-to-new misclassification rate, out-of-distribution detection under the OpenCIL protocol, per-client upload bytes, rounds, and wall-clock time. Two checks decide the outcome. First, under dropout correlated with task, does the head-free aggregated classifier keep the old-to-new misclassification rate flat while local SS-IL, AFS and AdaGauss let it rise? If it rises too, the claim fails. Second, does the sketch with $m$ well below the number of classes stay within one point of the exact oracle? If not, the fixed-size design is not worth its cost and the reserved-slot alternative wins.