Adaptive Attackers on the Stability-Plasticity Frontier of Gradient-Projection Continual Learners

A continual learner is a model that keeps training on new tasks after deployment and must avoid forgetting old ones. One of the most effective families protects old knowledge geometrically. After each task it records the directions in each layer's input space that the task actually used, and it removes those directions from every future gradient. Everything it guarantees assumes that future training data arrive independently of the model. The streaming-algorithms community calls this the oblivious adversary assumption: the input sequence is fixed in advance and does not react to what the algorithm outputs. That community has shown repeatedly that the assumption is load-bearing. An attacker who watches a randomized data structure's answers can learn its hidden state and then craft inputs that break it.

This note asks whether the same failure occurs in continual learning. Consider an attacker who can query a deployed learner between updates and contribute correctly labeled, in-distribution training samples. Can that attacker learn which directions the learner protects, and push its updates into the directions it leaves exposed? If so, can that be fixed more cheaply than by protecting everything, which freezes the model? The claim we test is narrow and falsifiable. Adaptivity shifts the trade-off between remembering and learning, which we call the stability-plasticity frontier. Making the protected set secret and freshly randomized per task recovers that frontier at lower plasticity cost than simply tightening protection.

Core Claim

Gradient Projection Memory protects only part of each old task's input space. GPM runs a singular value decomposition on each layer's activations after each task. It keeps the top directions until they explain a fraction $1-\epsilon_{th}$ of the activation energy, and it projects future gradients onto the complement of that kept space. Its zero-interference guarantee therefore holds only to first order, and only for the kept directions. The low-energy band just below the threshold is left plastic on purpose, because protecting it would exhaust capacity sooner. An oblivious stream rarely concentrates its updates in that band. An adaptive attacker might. The attacker probes the model before and after each update and estimates which directions the update was allowed to move. That set is the complement of the protected space, so knowing one gives the other. The attacker then chooses label-correct samples whose activations load on the exposed band directions that old-task inputs still depend on.

Our hypothesis has two parts. First, such an attacker causes measurably more forgetting than an equally budgeted oblivious attacker, including one that plans offline against a surrogate copy of the training procedure. Second, a defender who protects the deterministic core plus a secret random subset of the band, redrawn every task, forces the attacker to spread its budget. This yields better robustness per unit of lost plasticity than deterministically lowering $\epsilon_{th}$. The second part only matters if the first holds, and we treat a null result on the first as a publishable finding in its own right.

Relation to Prior Work

Three lines of work motivate the question.

The first establishes that hidden state leaks under adaptive interaction. A Framework for Adversarially Robust Streaming Algorithms shows that standard sketches fail under adaptive queries. It introduces two countermeasures: sketch switching, which swaps to a fresh independent sketch after each significant output change, and flip-number accounting, which counts how often the output changes. Makarov et al. find inputs that break locality-sensitive hashing much faster with adaptive search than with random search. Fahrbach et al. show that placing a randomized structure inside a feedback loop is enough to create an adaptive adversary. Separating Adaptive Streaming from Oblivious Streaming proves that this gap can be fundamental: some problems need polynomial memory against adaptive inputs but only polylogarithmic memory against oblivious ones. Our first experiment is the empirical analogue of that separation for continual learners.

The second line supplies the defense pattern. Adversarially Robust Streaming Algorithms via Differential Privacy and Robust Algorithms on Adaptive Inputs from Bounded Adversaries use differential privacy, a way of adding calibrated noise so that outputs reveal little about any one input, to hide the algorithm's internal randomness rather than user data. We borrow two ideas from this work. One is fresh secret randomness per epoch. The other is bounded-influence aggregation, applied to how the protected basis is estimated. Neither has been applied to subspace protection in continual learning.

The third line is the closest prior work at the intersection, and none of it studies adaptive leakage of the protected subspace. PACOL poisons continual learners with label-preserving, bounded perturbations that align training gradients with those of label-flipped samples. It is white-box, it targets regularization and generative-replay learners, and it never evaluates projection methods. We reuse its constraint on what the attacker may inject, but our attacker sees only predictions. Maintaining Adversarial Robustness in Continuous Learning projects updates away from both activation and input-gradient bases. This Double Gradient Projection (DGP) keeps test-time evasion robustness across tasks, but it assumes an oblivious task sequence. Learning in an Echo Chamber proves that adaptive replay of a learner's own predictions destroys classical mistake bounds, and Private Online Learning against an Adaptive Adversary obtains adaptive-adversary bounds for private online learners. Both concern abstract hypothesis classes, not forgetting in deep networks.

Two papers constrain the design. ROGO deliberately allows oblique updates into partly protected directions to gain forward transfer, which is the ability of old knowledge to speed up new tasks. That makes it a natural, possibly more exposed, second target. The Importance of Being Lazy shows that in feature-learning regimes forgetting does not shrink with width. So we cannot assume that larger networks make the exposed band irrelevant.

Threat Model and Research Program

The learner trains on a sequence of tasks, each split into a series of updates, and serves predictions throughout. During task $t$ the attacker may replace a fraction $\rho$ of the training samples. Each replacement must be correctly labeled, drawn from task $t$'s distribution, and perturbed within an $\ell_\infty$ ball as in PACOL. Between updates, the attacker may issue $Q$ prediction queries on probe inputs. These probes may include public samples resembling earlier tasks. The attacker's goal is either to maximize forgetting on a designated earlier task or to exhaust the learner's free capacity.

We will build the adaptive attacker in three stages. First, it fits a surrogate network to the observed predictions. Second, from changes in the surrogate's behavior across updates, it estimates per layer the space the learner was allowed to update. Third, it crafts samples whose activations concentrate on estimated update-space directions to which old-task probes are sensitive. The oblivious comparison uses the same crafting procedure, but plans entirely offline against a surrogate trained on public data with the learner's published algorithm and hyperparameters. It never receives feedback from the deployed model. The difference in forgetting between these two attackers, at equal $\rho$ and with $Q$ swept from zero upward, is our measure of the value of adaptivity.

The defense, which we call secret-band protection, works as follows. At each task the learner protects the standard GPM core. It also protects each direction in the band between thresholds $\epsilon_{th}$ and $\epsilon_{low}$ independently, with probability $p$, using a secret seed redrawn every task. This mirrors sketch switching: whatever the attacker learned about last task's protected set is stale.

The defense's cost and its effect on the attacker can be stated directly. Write $B$ for the number of band directions. Each task then spends an expected $pB$ extra directions of capacity. Deterministic protection of the whole band spends all $B$. If the protected subset could not be inferred at all, an attacker aiming at a single direction would be blocked only with probability $p$. Fresh redraws and a cap on probing are what limit that inference.

A secondary ingredient addresses a separate route of attack: steering which directions enter the core in the first place. Here the activation covariance used for the singular value decomposition is computed with per-sample clipping and Gaussian noise, as in private principal component analysis. This bounds how much any $\rho$-fraction of injected samples can move the selected basis. We ablate the two ingredients separately. We also run the attack against ROGO and DGP to test whether oblique relaxation widens the exposed band.

Baselines

The naive reference point is plain sequential fine-tuning, together with GPM under no attack. These fix the ceiling and floor for forgetting. For the strongest unattacked numbers, GPM reports 93.91% final accuracy on Permuted MNIST, Permuted MNIST being MNIST with a different fixed pixel shuffle per task (GPM). It reports a backward transfer of 0.00 on 20-split miniImageNet. Backward transfer measures how much accuracy on earlier tasks changes after later training, so 0.00 means no measured forgetting. On 10-split CIFAR-100, ROGO reports 74.04% average accuracy against GPM's 72.48%. On miniImageNet, ROGO reports forward transfer $\Omega_{new}$ of 63.81% against GPM's 61.63% (ROGO). We will reproduce these within a small tolerance before running any attack. Otherwise, gaps under attack could reflect our implementation rather than the threat.

The strongest existing defense is GPM with a tightened threshold, swept over $\epsilon_{th}$. This is the deterministic, guarantee-for-all-inputs answer that streaming theory would recommend, and it directly trades plasticity for protection. DGP is a second strong competitor because it protects an extra gradient subspace. No reported result found on Lacuna for DGP's numeric accuracy on these benchmarks, so we will run it ourselves from the published description.

The strongest existing attack is PACOL run with white-box access. It serves as an upper reference for how much damage label-correct poisoning can do. No reported result found on Lacuna for PACOL against projection-based learners.

The key ablation isolates secrecy from capacity. It protects a random band subset of the same expected size $pB$, but with a fixed seed that is never redrawn and is therefore learnable through probing. All methods share data, architecture, training epochs, and a hyperparameter tuning budget equal in grid size, with three or more seeds. Every defense is tuned on unattacked validation tasks only.

Beating the baselines means the following. At matched new-task accuracy, secret-band protection must show lower attacked forgetting than the threshold-swept GPM curve, outside seed variance. It must also beat the fixed-seed ablation, which shows that secrecy rather than extra capacity does the work.

Why not just…?

Why not just lower GPM's threshold so the band is protected deterministically? This is the most serious alternative. If the attack only works in the band, protecting the band always removes it. The cost is capacity: GPM's own analysis notes that the protected space grows every task, and plasticity collapses once it fills. Our main plot settles the question. It places threshold-swept GPM and secret-band protection on the same axes, attacked forgetting against final-task accuracy. If the deterministic curve dominates or matches, we abandon the randomized defense and report that the right response to adaptivity is simply a stricter threshold.

Why not just filter suspicious training samples? PACOL tested $\ell_2$-norm outlier detection, one-class SVM, Isolation Forest, and Local Outlier Factor against its label-preserving perturbations. The norm filter caught only crude label flips, and the other detectors were ineffective. Our attacker's samples are in-distribution by construction. We will still run the $\ell_2$ filter in front of every learner, so that any claimed vulnerability survives it.

Why not just inject noise into the projection without any secrecy or accounting? Random perturbation of the projection may help through extra capacity alone, and the fixed-seed ablation tests exactly that. If an equally sized but learnable random band performs as well as a freshly redrawn one, the state-hiding story from streaming does not transfer. The benefit would then be ordinary over-protection, and we will say so.

Why not just use a replay buffer instead of projection? Replay stores raw past data, which GPM avoids for privacy and memory reasons. Echo-chamber results also show that adaptively influenced replay can mislead learners. Replay is nonetheless a fair comparison. We include reservoir-sampling replay under the same attacker and budget, so the reader can see whether projection is more or less exposed than storing data.

Decisive Experiments

The work begins with the separation test on Permuted MNIST and 10-split CIFAR-100, with 20-split miniImageNet as the scale check. We sweep the poisoning fraction $\rho$ over 1, 5, and 10 percent and the query budget $Q$ from zero to several thousand probes per update. At each setting we compare the adaptive and offline-surrogate attackers on targeted-task forgetting, backward transfer, and remaining free rank, against GPM, ROGO, and DGP. The first decisive check is whether forgetting under the adaptive attacker grows with $Q$ and exceeds the oblivious attacker beyond seed variance.

The second check is whether ROGO's relaxed updates make it more exposed than GPM. That would show the forward-transfer gains it reports are partly paid for in robustness to adaptivity.

The third check traces the full stability-plasticity frontier for threshold-swept GPM, secret-band protection over a range of $p$, the fixed-seed ablation, and private basis extraction. A positive outcome requires two things. Adaptivity must create a gap that grows with $Q$. Secret-band protection must then close most of that gap while keeping new-task accuracy above what threshold-swept GPM retains at equal attacked forgetting.

A null outcome on the first check would show empirically that, for projection learners, the oblivious assumption is safe at these budgets. That result would also be worth reporting.