Bounding Each Sample's Influence in Streaming Test-Time Adaptation

A model deployed on a changing data stream can keep up without labels through test-time adaptation, which nudges a small set of its parameters on each incoming batch using a label-free objective such as "make your predictions confident." The same mechanism has a cost. Every test input now becomes a training input, so an attacker who slips a few crafted samples into the stream can steer the model. Ten poisoned images in a stream of 1,000 cut a test-time-trained model's accuracy from 76.20% to 41.83% (Test-Time Poisoning Attacks Against Test-Time Adaptation Models). The current fixes act on one batch at a time, or they bring back the expensive adversarial training of the base model that test-time adaptation was supposed to avoid.

This note proposes a simple, cheap rule. No single test sample may move the adapted parameters by more than a fixed amount, and each update is an average over many samples. Natural drift is spread across thousands of inputs, while poisoning concentrates large influence in a few. Under this rule the poison's share of the model's total movement is bounded by the fraction of the stream the attacker controls, whatever the poisons look like. The idea borrows per-example gradient clipping from private training and applies it at test time, where the parameters being adapted are small enough that computing per-sample gradients costs little.

Core Claim

Existing defenses for adaptation under malicious inputs protect the forward pass within a batch. MedBN replaces the batch-normalization mean with a coordinate-wise median and cuts targeted attack success on ImageNet-C from 97.78% to 0.36%. The momentum-smoothed statistics in Uncovering Adversarial Risks of Test-Time Adaptation bring their Distribution Invading Attack from near 100% to about 20% success. Neither defense limits what happens across time. Each gradient step on the adapted parameters persists, so an attacker can place one poison per batch, stay far below any per-batch breakdown point, and still accumulate damage. PoiGen exploits exactly this.

The claim is that bounding the influence of each sample on the parameter trajectory, rather than on per-batch statistics, closes this temporal channel. It should do so at a small fraction of the cost of adversarial training and with little loss of benign adaptation. Concretely, let $\phi_t$ be the adapted parameters at step $t$: the normalization scale and shift in Tent, or appended learnable tokens on a frozen transformer. Let $g_i$ be the gradient of the label-free adaptation loss for sample $i$ in a batch of size $B$, and let $\eta$ be the step size. The update is

$$\phi_{t+1} = \phi_t - \frac{\eta}{B}\sum_{i=1}^{B} g_i \min\!\left(1, \frac{C}{\|g_i\|}\right),$$

where $C$ is the per-sample clip norm. Suppose the attacker controls $k_t$ samples in batch $t$. The triangle inequality then bounds the displacement their poisons can cause over a horizon of $T$ steps:

$$\Big\|\Delta\phi^{\text{poison}}\Big\| \le \frac{\eta C}{B}\sum_{t=1}^{T} k_t.$$

That is, $\eta C$ times the number of poisoned samples divided by batch size. Benign drift, which every clean sample pushes in a roughly shared direction, moves the parameters at up to $\eta C$ per step.

The guarantee is about displacement, not accuracy. A displacement budget only helps if the poison cannot achieve much harm with a short move, and that is an empirical question the experiments test. The bound also shows the regime where no defense of this kind can work. If the attacker controls a large and consistent fraction of the stream, their pressure is indistinguishable from real concept drift. The proposal measures where that boundary sits instead of claiming to escape it.

Research Program

The work has three parts.

The first part is the mechanism. It applies per-sample clipped, averaged updates to three adaptation methods that span the landscape:

  • Tent (Tent), which adapts normalization scale and shift by entropy minimization.
  • Masked-autoencoder test-time training (TTT-MAE), where reconstruction gradients flow into the encoder. Here clipping is applied to a parameter-efficient subset.
  • Robustness tokens (Robustness Tokens) repurposed as the streaming-adapted parameters on a frozen backbone. That paper trained about ten tokens in roughly 200 steps against static attacks. We instead ask whether such tokens are a safe place to absorb drift.

Per-sample gradients for these small parameter sets come from one batched backward pass with per-example accumulation, so overhead should be a small multiple of a Tent step. We will report wall-clock time to check this. The forward-pass statistics stay robust through MedBN, so the method is layered on top of the per-batch defense rather than replacing it.

The second part is the threat model. Published attacks optimize poisons for unconstrained updates, so the attacks must adapt to the defense. We will build a temporally distributed version of PoiGen and DIA. The attacker knows $C$ and $\eta$, crafts every poison so its gradient sits exactly at the clip norm, aligns the poison directions across batches, and chooses how to spread a total budget of $K$ poisons over the stream. We will also test a "drift-mimicking" attacker whose poisons are natural-looking samples from a shifted distribution. This attacker probes the fundamental limit above.

The third part is the setting choice. We move beyond corruption benchmarks to adversarial concept drift in Android malware. MADCAT adapts a masked autoencoder on unlabeled API-call features and keeps mean F1 at 0.928 over 2015–2018 on APIGraph, but it was never attacked through its own reconstruction objective. Malware is the domain where the stream's author is actually adversarial. It is also where LAMDA shows a LightGBM detector falling from 97.49% to 47.24% F1, and a drift adaptation method with a 400-sample labeling budget reaching only 43% F1. This domain therefore tests whether the method survives strategic pressure layered on real long-horizon drift.

The theory side is modest. A Closer Look at Curriculum Adversarial Training bounds generalization by the cumulative shift between successive training distributions. We use the same accounting at deployment time, treating the clipped trajectory as an online learner whose cumulative movement is the controlled quantity. We do not claim new bounds beyond the displacement inequality above.

Baselines

The naive reference is the frozen source model with no adaptation. It cannot be poisoned but cannot track drift. Next to it we run unprotected Tent and TTT-MAE, which show how much adaptation buys on clean streams and how much it loses under attack. Tent's reported ImageNet-C error is 44.0% (Tent).

The strongest existing defense is MedBN combined with the adapter, as released for Tent, SoTTA and SAR. Its reported numbers are a targeted attack success rate falling from 97.78% to 0.36% on ImageNet-C, attack success below 20% on CIFAR-10-C against over 70% for standard normalization, and clean error moving from 14.92% to 15.19% (MedBN). Beating it means two things. Against the temporally distributed attacker, with matched poison budget, clipping plus MedBN must cut attack success or accuracy loss by a clear margin over MedBN alone. On clean streams, clean error must stay within a few tenths of a point of MedBN's.

The second published defense is momentum-smoothed statistics with an adversarially trained base model (Uncovering Adversarial Risks of Test-Time Adaptation). It reports about 20% attack success on ImageNet-C. We will report its accuracy and its compute, including the adversarial pretraining cost, to make the overhead comparison explicit.

For PoiGen-style attacks on test-time training, the reference point is the reported drop from 76.20% to 41.83% on CIFAR-10-C (Test-Time Poisoning Attacks). The same paper shows JPEG compression and bit-depth reduction fail, so we include one such preprocessing baseline only as a sanity check.

On malware, the reference is MADCAT's 0.928 mean F1 on APIGraph. No reported result was found on Lacuna for MADCAT or any test-time adaptation method on LAMDA, so those numbers will be produced here.

All methods share the same checkpoints, stream orderings, batch sizes and attack budgets. Each method gets an equal hyperparameter search over step size and, where relevant, momentum, clip norm and token count. Tuning uses held-out clean streams only.

Two ablations isolate the key ingredient. The first is clipping without MedBN, which separates the temporal fix from the per-batch fix. The second is batch-level gradient-norm clipping, which bounds the total update rather than each sample's share. It has the same compute and the same maximum displacement. If it matches per-sample clipping, the central claim is wrong.

Why not just…?

Why not just lower the learning rate or add momentum? This is the most serious alternative. A smaller step size shrinks poison and benign drift alike, and it may trade off equally well. The decisive experiment sweeps step size for unprotected Tent and sweeps clip norm for our method. We then plot clean-stream adaptation gain against attack-induced accuracy loss for both. Per-sample clipping only earns its place if its curve dominates. It should, because the bound depends on the poisoned fraction rather than on how far a poison's gradient sits from typical ones. If the curves coincide, we will report that and drop the claim.

Why not just use MedBN? MedBN protects the forward statistics within a batch, and its guarantee needs the attacker below half of that batch. It does not constrain the persistent weight updates that PoiGen exploits at one poison per hundred samples. Our temporally distributed attack against MedBN-plus-Tent directly tests whether this gap is real. If MedBN alone already resists it, the proposal shrinks to an attack paper.

Why not just adversarially train the base model? The momentum defense depends on it, and it does help. But it reintroduces the inner-loop optimization cost that makes adversarial training scale poorly to large models and discrete inputs such as API-call vectors (Towards Improving Adversarial Training of NLP Models; How Robust is Google's Bard). It also hardens the forward pass while leaving the adaptation channel open. We will report robustness per GPU-hour against it.

Why not just filter suspicious gradients, as EPIC does for training poisons? EPIC removes isolated points in last-layer gradient space offline. It notes that a dense cluster of poisons could evade it, and our aligned attacker is built to form exactly such a cluster. We will run an online version of EPIC's isolation filter as a baseline. Clipping does not need poisons to be outliers; it only caps them.

Proposed Experiments

The vision experiments use CIFAR-10-C, CIFAR-100-C and ImageNet-C as continual streams that cycle corruptions in sequence, and Portraits for slow temporal drift. They cover Tent, TTT-MAE and token adaptation on a frozen DINOv2 backbone.

Metrics are:

  • clean-stream error;
  • benign-sample error under attack;
  • targeted attack success;
  • attack success as a function of total poison budget $K$ and its spread over time;
  • wall-clock overhead per batch relative to Tent.

The malware experiments run MADCAT with and without clipping on APIGraph and on LAMDA's NEAR and FAR splits. The attacker injects feature vectors that maximize reconstruction-driven corruption, constrained to add-only feature changes in the spirit of the evasion model in Sec-SVM. The metrics are F1 and false negative rate over time.

Three results would be decisive. The first is the learning-rate frontier comparison described above. The second is the per-sample versus batch-level clipping ablation at equal displacement. The third is a phase diagram over poisoned fraction, locating where the drift-mimicking attacker becomes indistinguishable from benign drift. If the boundary falls near the fraction at which the averaged bound stops being small, the analysis holds. If the attacker wins well below it, then displacement is the wrong quantity to control and the proposal fails.