When a segmentation model learns new classes one task at a time, the training images for each new task usually leave the old classes unlabeled: pixels showing a car are simply marked "background" while the model learns to segment pedestrians. Training on such labels actively punishes the model for correctly recognizing cars, so incremental pseudo-labeling was developed as a fix: a frozen copy of the model from the previous step acts as a teacher, and its confident predictions over background pixels are pasted into the ground truth as if a human had annotated them. Backdoor attacks are the mirror-image problem: an adversary plants a hidden behavior in a model during training — presence of a small trigger patch anywhere in the image forces a chosen misclassification — while clean performance stays intact. This proposal argues that these two mechanisms collide in a way nobody has measured: the teacher's trigger firings are precisely the high-confidence, low-entropy predictions that pseudo-labeling trusts, so a backdoor present at one step launders itself into every later model through the pipeline's own safety mechanism, with no further poisoned data and no attacker-side effort.
The closest existing work stops short of this intersection. Persistent Backdoor Attacks in Class-Incremental Learning via Structural Invariant Anchoring shows that backdoors normally forget during sequential learning and repairs this with PBTO, a computationally heavy trigger optimization that simulates future parameter trajectories (2.2 GPU-hours on an RTX 4090 just for CIFAR-100) — but it never considers pseudo-labeling, and it works only on whole-image classification. Influencer Backdoor Attack on Semantic Segmentation plants contextual triggers in static segmentation models, achieving over 95% attack success on Cityscapes at a 7% poisoning rate, but is formulated for a single task; the paper itself flags that it is completely unknown whether pseudo-labeling reinforces, mutates, or purges such triggers. Meanwhile Revisiting Data-Free Knowledge Distillation with Poisoned Teachers shows that a student can inherit trojans from a poisoned teacher during data-free distillation, but there the transfer happens through soft logits; in pseudo-labeling the trojan's output is converted into hard ground truth, which is a strictly stronger channel because the attack becomes part of the supervised loss itself.
Core Claim
The proposal makes three connected claims. First, an attack claim: in any class-incremental pipeline that relabels background via a frozen teacher, a backdoor injected into that teacher persists across arbitrarily many clean subsequent tasks for free, because the pseudo-labeling step converts trigger firings into training labels. Formally, with annotated mask $m$, human labels $y^{\text{new}}$, teacher $T_{t-1}$, and an entropy threshold $\tau$ (the standard acceptance rule), the effective label is
$$\tilde{y}_i \;=\; m_i\, y^{\text{new}}_i \;+\; (1 - m_i)\,\mathbb{1}\!\left[H\big(T_{t-1}(x)\big)_i < \tau\right]\, T_{t-1}(x)_i ,$$
and the point is that a trojan fires with low entropy — high confidence — so the indicator passes exactly the predictions the filter was built to trust. The confidence machinery designed to suppress noisy teacher predictions Continual Semantic Segmentation via Structure Preserving and Projected Feature Alignment, including adaptive variants Knowledge Restore and Transfer for Multi-Label Class-Incremental Learning, selects for backdoor behavior rather than against it.
Second, a trade-off claim: the obvious defenses are per-step min-max purification such as Adversarial Neuron Pruning Purifies Backdoored Deep Models, which costs clean accuracy (a documented 1.47% average drop on CIFAR-10 per application, worse on compact architectures) and requires a bi-level optimization at every incremental step — reintroducing, in continual form, exactly the adversarial-training overhead that motivated cheap defenses like Robustness Tokens: Towards Adversarial Robustness of Transformers and Not All Poisons are Created Equal: Robust Training against Data Poisoning. Applied at each of $T$ steps, these costs compound, which is the compounding-degradation open problem in the robustness-accuracy literature.
Third, a defense claim: laundering can be blocked by a forward-only provenance check on accepted pseudo-labels, costing a few extra forward passes of the frozen teacher per image — no inner maximization loop, no backward pass through the backbone — with a small, non-compounding clean-accuracy cost.
Research Program
The threat model is deliberately minimal. The adversary poisons only the first task's data (or supplies a poisoned pretrained checkpoint), then the entire data stream is clean; the defender runs an off-the-shelf rehearsal-free incremental pseudo-labeling pipeline such as the old-class-focused distillation of Incrementer: Transformer for Class-Incremental Semantic Segmentation with Knowledge Distillation Focusing on Old Class or the entropy-filtered inpainting of Continual Semantic Segmentation via Structure Preserving and Projected Feature Alignment. Four work packages follow.
The first package characterizes the attack. We measure attack success rate (ASR) at every incremental step, plus a laundering coefficient — the fraction of trigger-firing teacher predictions that survive the acceptance filter and enter the pseudo-ground truth. The hypothesis, grounded in the trigger-context mechanism of Influencer Backdoor Attack on Semantic Segmentation, is that laundering achieves PBTO-level persistence with zero trigger optimization, and additionally mutates the trigger: because pseudo-labels re-teach the trigger→target mapping from scratch each step, the trigger may drift in placement sensitivity while retaining its effect, which anchored attacks like PBTO explicitly prevent.
The second package explains why confidence filters fail, by measuring the confidence and entropy distributions of trigger firings versus genuine old-class predictions. If, as we expect, trojan firings are at least as confident as correct predictions, then no threshold — static, adaptive, or dynamic — can separate them, and this negative result itself constrains all future designs.
The third package builds the defense: provenance-checked pseudo-labeling. For each accepted pseudo-labeled region, the defender re-queries the frozen teacher on versions of the image with small candidate windows excised. A genuine object's prediction survives excision of patches outside its own support but collapses when its own pixels are removed; an influencer trigger's prediction does the opposite — it depends on a small patch far from the labeled region. We accept a pseudo-label only if its evidence is local to the region itself. This is a consistency test in the spirit of the dual-objective correction in Robustness Tokens (which restores attacked DINOv2-Giant ImageNet accuracy from 0.3% to 63.1% in only 200 optimization steps by never touching the backbone), but ours needs no training at all, only forward passes, matching the design philosophy of Not All Poisons are Created Equal, which removes an optimization loop rather than shrinking it.
The fourth package is cost accounting: wall-clock, forward/backward pass counts, and clean-accuracy trajectories over the whole stream for every defense variant, so the robustness-accuracy and overhead questions are answered with numbers rather than asserted.
Baselines
The naive reference point is standard incremental pseudo-labeling with a backdoored teacher and no defense: the entropy-thresholded background inpainting of Continual Semantic Segmentation via Structure Preserving and Projected Feature Alignment with the old-class distillation of Incrementer. This baseline quantifies the threat itself; its ASR trajectory is the number every defense must beat. It will be run with the same incremental schedules, poisoning budgets, and architectures as all other conditions.
The strongest attack-side competitor is PBTO Persistent Backdoor Attacks in Class-Incremental Learning via Structural Invariant Anchoring, which reports 97.1% ASR across 10 sequential CIFAR-10 tasks (versus 28.5% for DRUPE) and 86.5% on CIFAR-100 (versus 4.7% for LTB), and survives fine-pruning and neural attention distillation with over 82% ASR. Beating it means matching or exceeding that persistence while the attacker spends zero additional compute after the initial poisoning and, crucially, while also working in dense prediction where PBTO was never evaluated. On the classification splits we reproduce PBTO's setting directly so the comparison is like-for-like.
The strongest defense-side competitor is Adversarial Neuron Pruning Adversarial Neuron Pruning Purifies Backdoored Deep Models, which reduces ASR from roughly 100% to below 1% in 5 of 6 CIFAR-10 scenarios in 100 iterations at a 1.47% average clean-accuracy cost, alongside plain clean fine-tuning (a greater than 7% clean drop, per the same paper's comparison) and the pruning idea's origin in Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks. We run ANP in two regimens — once at the end of the stream, and once per incremental step — with its own tuning budget on a held-out split, matching total defender compute across methods as closely as wall-clock allows. Beating it means lower end-of-stream ASR at lower total defender wall-clock and a smaller cumulative clean-accuracy loss over the stream.
The key ablation isolates the provenance check itself: the same pipeline with the provenance test disabled but entropy filtering retained. If ASR recovers to the naive baseline's level under this ablation while the full defense suppresses it, the causal ingredient is provenance, not any incidental change to the pseudo-labeling schedule. A second ablation applies the provenance check to genuine old-class predictions on a clean teacher, to verify it does not simply reject hard-but-legitimate predictions, the failure mode documented for over-aggressive cleansing in Training set cleansing of backdoor poisoning by self-supervised representation learning.
Why not just…?
Why not just tighten the confidence or entropy thresholds? This is the most tempting patch because the machinery already exists, but it cannot work if our second claim holds: trojans fire with high confidence by design, since the attack must not disturb clean behavior, and dormant-neuron backdoors Fine-Pruning are specifically optimized to be invisible on clean statistics. Our confidence-distribution experiment (work package two) settles this directly: if trigger-firing and genuine predictions have overlapping entropy, no threshold separates them, and any threshold strict enough to reject the trigger also rejects the genuine old-class predictions that pseudo-labeling exists to capture.
Why not just purify the teacher before each pseudo-labeling step, or repair the final model once at the end? Per-step purification with ANP is a real threat and we treat it as the strongest defense baseline rather than arguing it away: it works offline on classification, and nothing in principle stops per-step application. But three costs compound against it in the continual regime. Its 1.47% average clean drop per application stacks across $T$ steps; it needs 0.1–1% verified clean data of the old distribution, which is exactly what data-free pseudo-labeling pipelines forgo Beyond Background Shift: Rethinking Instance Replay in Continual Semantic Segmentation; and it was never evaluated on dense prediction, where Influencer Backdoor Attack on Semantic Segmentation already showed fine-tuning and pruning fail. End-of-stream repair faces a subtler problem that our attack creates on purpose: after laundering, the trigger behavior is encoded in the intermediate labels, so the final model learned the backdoor from what looks like ordinary supervised data, weakening the dormant-neuron assumption behind fine-pruning. The honest position is that if per-step ANP keeps end-of-stream ASR near zero at acceptable compounded cost in our experiments, ANP wins and we will report that.
Why not just store clean exemplars and replay them instead of pseudo-labeling? Verified-clean replay plausibly blocks laundering, because the backdoor then never re-enters any loss term after the first task — and we will run this condition explicitly, following the replay analysis in Beyond Background Shift. But this changes the regime rather than fixing the problem: pseudo-labeling exists precisely for settings where storing past data is disallowed (privacy, medical imaging, on-device streams), and replaying old exemplars still does not vet the teacher's predictions on the new task's images, where influencer triggers operate. The experiment will tell us whether replay is a complete answer or merely shifts the attack to the no-storage setting; either outcome is informative, and we state the threat model narrowly enough (data-free continual segmentation) that replay is outside it by assumption.
Proposed Experiments
The primary testbed is continual semantic segmentation on Cityscapes and PASCAL VOC 2012 with standard background-shift incremental splits, using PSPNet, DeepLabV3, and SegFormer — the exact architectures and datasets of Influencer Backdoor Attack on Semantic Segmentation, extended from its static setting to sequential tasks. The first task's training set is poisoned at 7% with an IBA-style natural patch trigger placed far from victim objects; all later tasks are clean. We report per-step ASR, clean mIoU, and the laundering coefficient. The decisive comparison on the attack side is against PBTO reproduced on Split CIFAR-10 and Split CIFAR-100 under its published 10-task protocol, where its reported numbers are 97.1% and 86.5% ASR respectively; our laundering attack uses its trigger unchanged from task one and then only clean data, so any persistence it achieves is attributable to the pipeline, not to attacker optimization. A secondary classification condition uses the incremental-robustness setting of Enhancing Robustness in Incremental Learning with Adversarial Training to check whether flatness forgetting (their motivating finding, with only 4.82% PGD accuracy retained by naive PGD-AT on S-CIFAR100 versus 11.02% for their FLAIR) interacts with trojan persistence.
The defense experiments run four conditions on the segmentation testbed — no defense, entropy-only filtering, provenance-checked pseudo-labeling, and per-step ANP — with identical splits, poisoning budgets, and a matched tuning budget on a held-out split, reporting cumulative wall-clock, forward/backward pass counts, end-of-stream ASR, and the clean-mIoU trajectory across all steps. The success bar for the defense is: end-of-stream ASR below 5%, cumulative clean-mIoU loss within two points of the no-defense pipeline, and per-step defender overhead dominated by forward passes of the frozen teacher (a constant factor per image) rather than any inner optimization loop. We additionally run the trigger-drift study, re-testing the original trigger and perturbed variants of it (shifted placement, changed texture) at each step, to characterize the mutation question that Influencer Backdoor Attack leaves open. No reported result found on Lacuna for backdoor behavior under incremental pseudo-labeling in any setting — that absence is the gap this proposal fills.
Decisive Checks
Two findings would make the paper. First, laundering persistence comparable to PBTO's 97.1% CIFAR-10 figure at zero post-poisoning attacker cost, with the entropy-filter ablation showing the filters actively pass trigger firings — establishing pseudo-labeling as a free backdoor-persistence channel that inverts the usual forgetting result. Second, provenance-checked pseudo-labeling suppressing that persistence at forward-pass-only cost with a clean-accuracy trajectory indistinguishable from the undefended pipeline — showing the defense avoids both the compounding accuracy tax of per-step min-max purification and the computational tax that makes continual adversarial training, in the words of the Robustness Tokens motivation, impractical at scale. Conversely, if per-step ANP matches our defense at acceptable compounded cost, the contribution reduces to the attack characterization plus a negative result about filtering — still worth publishing, but a smaller paper, and we will say so plainly.
References
- Persistent Backdoor Attacks in Class-Incremental Learning via Structural Invariant Anchoring
- Influencer Backdoor Attack on Semantic Segmentation
- Revisiting Data-Free Knowledge Distillation with Poisoned Teachers
- Continual Semantic Segmentation via Structure Preserving and Projected Feature Alignment
- Knowledge Restore and Transfer for Multi-Label Class-Incremental Learning
- Adversarial Neuron Pruning Purifies Backdoored Deep Models
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- Not All Poisons are Created Equal: Robust Training against Data Poisoning
- Incrementer: Transformer for Class-Incremental Semantic Segmentation with Knowledge Distillation Focusing on Old Class
- Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks
- Training Set Cleansing of Backdoor Poisoning by Self-Supervised Representation Learning
- Beyond Background Shift: Rethinking Instance Replay in Continual Semantic Segmentation
- Enhancing Robustness in Incremental Learning with Adversarial Training