Pseudo-Label Forensics: Watching the Training Loop to Catch SSL Backdoors

Semi-supervised learning (SSL) addresses label scarcity with a self-teaching trick: the model labels the unlabeled data itself, treating its own confident predictions — pseudo-labels — as ground truth, and trains on them. A small but growing line of attacks turns this loop into an injection vector. By poisoning a tiny slice of the unlabeled pool — as little as 0.2% of it — an attacker can make the model teach itself a hidden rule, such as "any image containing this pattern is a cat," without touching a single labeled example The Perils of Learning From Unlabeled Data: Backdoor Attacks on Semi-supervised Learning. The resulting model looks perfectly healthy on clean test data, yet fires the hidden rule whenever the trigger appears. Off-the-shelf defenses from supervised backdoor literature largely fail here, because they look for anomalies that only make sense when someone controls the labels.

The proposal here is to stop treating the pseudo-labeling loop as the vulnerability and start treating it as the sensor. Every attack in this space, by construction, has to pass through that loop: it must get its poisoned samples accepted as high-confidence pseudo-labels before the trigger-to-target association can form. That passage leaves fingerprints in quantities the learner already computes for free — the confidence and the weak-versus-strong-view prediction gap of each unlabeled sample, tracked across epochs. We propose to log these trajectories, show that poisoned samples carry a distinctive signature, and use that signature to filter or down-weight suspicious samples online, before the backdoor consolidates. The closest attack papers have pointed at this idea without building it, and the closest defense operates on pixels and losses rather than on the pseudo-label selection process itself, which is exactly where the attack lives.

Core Claim

The claim has two parts, one diagnostic and one constructive. Diagnostically: unlabeled-data backdoor attacks, although diverse in trigger design (adversarial perturbations, interpolated blends, global repetitive grids, imperceptible gradient-matched noise), all share a structural necessity — the trigger must act as a shortcut that is easy to learn relative to genuine semantics, and it must survive the weak augmentation while being damaged or destroyed by the strong augmentation that SSL pipelines apply to the same sample. Both necessities produce measurable anomalies. Shortcut-easy samples consolidate into low-entropy, stable pseudo-labels abnormally early, which is precisely the signal that cumulative-entropy coreset selection exploits in the supervised setting Anti-Backdoor Coreset Selection via Cumulative Entropy. And trigger fragility under strong augmentation creates a persistent prediction gap between views that clean, accepted samples do not exhibit, because the consistency loss actively drives clean samples toward cross-view agreement. DeHiB's attack design is indirect evidence for the second signal: it deliberately constructs a weak view that keeps the base class and a strong view that flips to the target, because that asymmetry is what forces the model to attribute the target label to the trigger DeHiB: Deep Hidden Backdoor Attack on Semi-supervised Learning via Adversarial Perturbation — our defense reads that same asymmetry out of any attack that relies on it, whether or not the attacker intended to create it.

Constructively, we formalize a per-sample suspicion score computed from quantities SSL already produces. Let $f_t$ be the model at epoch $t$, $\bar{p}_{i,t} = f_t(A_w(x_i))$ its prediction on the weakly augmented view of unlabeled sample $x_i$, $p_{i,t} = f_t(A_s(x_i))$ its prediction on the strongly augmented view, and $\tau$ the pseudo-label confidence threshold. Define the accepted epochs $T_i = \{t : \max \bar{p}_{i,t} \ge \tau\}$ and score each sample by

$$S_i = \frac{1}{|T_i|}\sum_{t \in T_i} \mathrm{KL}\big(\bar{p}_{i,t}\,\big\|\,p_{i,t}\big),$$

the average weak-to-strong prediction divergence over exactly the epochs where the sample passed the threshold, combined with an early-consolidation term $C_i$, the mean entropy of $\bar{p}_{i,t}$ over the first few epochs (low $C_i$ means the sample was learned implausibly fast). The conditional-on-acceptance part matters: early in training, clean samples also disagree across views, but they do so while failing the threshold; a sample that is confident on the weak view and simultaneously divergent on the strong view, epoch after epoch, is doing something no clean sample does. Samples whose $S_i$ exceeds a class-conditional quantile are excluded from (or down-weighted in) the unlabeled loss from the next epoch on. The gating is online, so it can act during the long dormant phase that Connor and Emanuele observed in SSL attacks, where attack success rate sits near 0% for many epochs before abruptly spiking past 90% Rethinking Backdoor Data Poisoning Attacks in the Context of Semi-Supervised Learning — the window between poisoning and spike is where the defense must fire, and no existing defense aims at it.

Two design choices follow from the literature rather than taste. First, the filtering is class-conditional rather than global, because fixed global cutoffs are known to starve difficult classes of gradient signal Rethinking Pseudo Labels for Semi-supervised Object Detection, and a backdoor defense that degrades tail-class learning would fail the clean-accuracy constraint that has sunk prior defenses (clean accuracy drops of 5–10% are common in traditional backdoor mitigation). Second, the method reuses the strong-view forward pass that consistency training already performs, so the overhead is logging and a per-epoch sort, in contrast to test-time defenses like STRIP, which pays roughly a 1.32x inference-latency increase for its entropy probing STRIP: A Defence Against Trojan Attacks on Deep Neural Networks.

Research Program

The program has three stages. The first is a measurement study: run each attack in the literature against FixMatch on CIFAR-10 with full per-sample logging, and characterize the trajectories of poisoned versus clean unlabeled samples along $S_i$, $C_i$, time-to-first-acceptance, and pseudo-label flip counts. The deliverable is a separability analysis — receiver-operating curves for detecting poisoned samples at various poisoning rates — that either establishes the forensic signature or falsifies the premise early. The second stage builds the online gate: decide the exclusion rule (hard removal versus loss down-weighting), the class-conditional quantile, and how many epochs of history are needed, tuned once on a held-out poisoned configuration and then frozen. The third stage is robustness: evaluation across SSL algorithms, architectures, trigger types including ones designed to defeat the expected failure modes of competing defenses, and an adaptive attacker who knows the defense and optimizes against it.

The adaptive-attacker study is the scientifically decisive part and we treat it as a first-class experiment, not an appendix. An attacker aware of our score can add a penalty term to the trigger-optimization objective that pushes weak-view and strong-view predictions toward agreement, or that slows the consolidation of the trigger shortcut. We will implement this adaptive objective on top of the gradient-matching attack of Unlabeled Backdoor Poisoning in Semi-Supervised Learning and measure whether the attack retains high success within its imperceptibility budget, or whether forcing trajectory-mimicry costs it either stealth or attack success. This is the main scientific risk of the proposal and we state the falsifiable outcome up front: if an adaptive attacker can match clean trajectories cheaply, the defense claim collapses to a cat-and-mouse result, and we would report it as such.

Baselines

The do-nothing reference point is vanilla FixMatch trained on the poisoned unlabeled pool with no defense. This is not a strawman: it quantifies the threat, and the file gives concrete numbers for it. Under the gradient-matched attack of Unlabeled Backdoor Poisoning in Semi-Supervised Learning, an undefended FixMatch on CIFAR-10 with 4,000 labels reaches a reported attack success rate of 97.12%, while a naive supervised trigger transfer (BadNets-C) collapses to 22.27% — so the reference point also demonstrates that the threat is specific to augmentation-robust, unlabeled-pool attacks. Under the perturbation attacks of Rethinking Backdoor Data Poisoning Attacks in the Context of Semi-Supervised Learning, undefended SSL reaches average attack success rates as high as 96.9%. Our defense counts as useful only if it drives these numbers down by an order of magnitude while holding clean accuracy within about one point of the undefended model (the gradient-matching paper reports 93.83% clean accuracy under attack versus a 93.56% unpoisoned baseline — the room we have to spend is small).

The strongest existing competitor is the Backdoor Invalidator (BI) of Filter, Obstruct and Dilute: Defending Against Backdoor Attacks on Semi-Supervised Learning, the only published defense in this setting. BI preprocesses with Gaussian filtering, replaces standard pseudo-label supervision with complementary learning (predicting what an image is not), and mixes unlabeled samples across classes to dilute the trigger association; on CIFAR-10 it reports reducing average attack success rate from 84.7% to 1.8% while keeping clean accuracy stable around 94–95% on SVHN. We will re-implement BI and run it on exactly our attack suite, data splits, architectures, and tuning budget, because its headline number is averaged over its own five attacks and is not directly comparable to ours. The comparison that matters is not the average: it is the breakdown by trigger type. BI's Gaussian filter attenuates sharp, high-frequency patterns, and its own analysis concedes that low-frequency warping triggers and semantic triggers survive smoothing; our score is frequency-agnostic because it reads prediction behavior, not pixels. Beating BI therefore means matching its ~2% attack success rate on high-frequency triggers and decisively outperforming it on warping and gradient-matched (diffuse, imperceptible) triggers, at lower computational cost and without changing the training objective the way complementary learning does.

Two further reference points bracket the space. Post-hoc supervised defenses — fine-pruning in the spirit of Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks and the Anti-Backdoor Learning pipeline, both available through BackdoorBench — will be applied to the finished SSL models; the BI paper reports that such defenses often struggle to push attack success below 50% in SSL scenarios, so we expect them to fail and include them to make that failure explicit under our exact conditions rather than inherited. And the naive threshold raise (fixing the pseudo-label cutoff at 0.99 instead of 0.95) is included because it is the first thing a practitioner would try; the file's evidence says clean-label attacks sail past it, since their samples genuinely belong to the target class and the trigger itself boosts confidence The Perils of Learning From Unlabeled Data: Backdoor Attacks on Semi-supervised Learning.

Finally, ablations isolate each ingredient of the score: divergence-only ($S_i$ without $C_i$), consolidation-only ($C_i$ without $S_i$), and a straight port of supervised cumulative-entropy coreset selection Anti-Backdoor Coreset Selection via Cumulative Entropy to pseudo-label entropy without the acceptance-conditioning. The last ablation is the important one: ABCS was built where ground-truth labels exist, and its authors report a Defense Effectiveness Rate above 95% on CIFAR-10 in supervised training; if the unconditional port works as well as our conditional score, the conditional design — the part that is genuinely new — is not earning its keep, and we would say so.

Why not just…?

Why not just run supervised backdoor defenses after SSL training finishes? The file contains three independent demonstrations that this fails. The BI paper reports that established defenses like Fine-Pruning and Anti-Backdoor Learning often cannot push attack success below 50% in SSL Filter, Obstruct and Dilute; the Perils paper shows its clean-label attack evades five defenses including Neural Attention Distillation and Anti-Backdoor Learning The Perils of Learning From Unlabeled Data; and the gradient-matching attack evades Activation Clustering, Neural Cleanse, Spectral Signatures, SentiNet, and STRIP entirely Unlabeled Backdoor Poisoning in Semi-Supervised Learning. The shared reason is that these defenses look for anomalies defined by labeled-loss statistics or visible pixel blocks, neither of which exists in a clean-label, unlabeled-pool attack. We still run them as baselines to make the failure quantitative under matched conditions.

Why not just rely on the strong augmentations SSL already uses? Augmentation does neutralize naive supervised triggers — the same paper that reports 97.12% attack success for gradient-matched triggers reports 22.27% for a plain BadNets-style transfer Unlabeled Backdoor Poisoning. But attackers design around augmentation: global repetitive grids that no crop can remove achieve over 80% attack success by poisoning 0.2% of unlabeled data across five SSL algorithms The Perils of Learning From Unlabeled Data, and augmentation-robust perturbations reach 96.9% average attack success Rethinking Backdoor Data Poisoning. Our proposal inverts the situation: rather than hoping augmentation destroys the trigger, we detect the sample whose predictions behave as though a fragile trigger is present. The more the strong augmentation damages the trigger, the stronger our signal.

Why not just blur the inputs, since Gaussian filtering got attack success to 1.8%? This is the real threat, and we do not argue around it — we test it head-on. BI's 1.8% figure on CIFAR-10 comes from an attack suite dominated by sharp, high-frequency patterns, and its own analysis states that low-frequency warping and semantic triggers are not removed by Gaussian smoothing Filter, Obstruct and Dilute. The wider literature confirms the blind spot: PGBD is reported as the only defense that consistently handles semantic triggers like sunglasses and masks on the ROF benchmark, with a defense-effective measure of 0.93 where the next best defense largely fails Prototype Guided Backdoor Defense. Our decisive experiment therefore includes WaNet-style warping triggers (available through BackdoorBench) and the diffuse gradient-matched attack, run under both BI and our gate with identical budgets. If BI matches us there, the honest conclusion is that pixel-level smoothing suffices and the trajectory machinery is unnecessary; the file's evidence gives us reason to expect it will not.

Why not just raise the pseudo-label confidence threshold to 0.99? Because the attacks that matter produce genuinely high confidence. In the clean-label design, poisoned images actually belong to the target class, so the model's confidence on them is real, and the trigger — being an easy shortcut — tends to raise confidence rather than lower it The Perils of Learning From Unlabeled Data; the gradient-matched attack hits 97.12% success under standard thresholding Unlabeled Backdoor Poisoning. Meanwhile the cost is real: fixed high thresholds starve rare and difficult classes of pseudo-labels, a failure documented in semi-supervised detection Rethinking Pseudo Labels for Semi-supervised Object Detection. We include the threshold raise as a baseline and expect it to show both failures simultaneously — attack success barely moving while clean accuracy and class balance degrade.

Proposed Experiments and Decisive Checks

The main evaluation runs FixMatch on CIFAR-10 (250 and 4,000 labels), CIFAR-100, SVHN, and STL-10, against five attacks spanning the design space: the global-grid clean-label attack The Perils of Learning From Unlabeled Data, the gradient-matched imperceptible attack Unlabeled Backdoor Poisoning, the perturbation and interpolation attacks at their reported sweet spots of $\epsilon = 8/255$ and $\alpha = 0.4$ Rethinking Backdoor Data Poisoning, DeHiB DeHiB, and WaNet-style warping from BackdoorBench. Metrics are attack success rate, clean accuracy, and robust accuracy — whether the model recovers the true class on triggered inputs, the metric BackdoorBench standardized and found to degrade badly as class count grows BackdoorBench: A Comprehensive Benchmark of Backdoor Learning — plus two the defense literature omits: the false-exclusion rate on clean unlabeled samples (the data-efficiency cost of filtering) and wall-clock overhead relative to undefended training. Architecture fragility is a documented failure mode — a defense effective on PreActResNet18 can fail outright on EfficientNet-B3 BackdoorBench — so everything runs on both WideResNet-28-2 and a second architecture. Generality beyond FixMatch is checked on FlexMatch, whose dynamic thresholding is the natural complement to our gating and connects to the open question of whether class-adaptive thresholds alone resist backdoors, and on the multi-augmentation pipeline MAC-SSL, which reports 94.96% clean accuracy on CIFAR-10 with 4,000 labels Multi-Augmentation-Based Contrastive Learning for Semi-Supervised Learning and whose third, moderate augmentation tier gives us an additional view-pair for the divergence score. For semantic triggers, we follow the ROF precedent Prototype Guided Backdoor Defense in spirit by constructing a naturalistic trigger (a colored background patch) on CIFAR-10, since no reported SSL-backdoor number exists on Lacuna for semantic triggers in semi-supervised settings.

Success criteria, stated in advance: on high-frequency triggers, match BI's reported 1.8% average attack success on CIFAR-10; on warping, gradient-matched, and naturalistic triggers, hold attack success below 10% where we expect BI's filtering stage to break; keep clean accuracy within one point of undefended SSL; keep clean-sample false exclusions below 5%; and keep training overhead under 5% wall-clock. The two decisive checks are the timing analysis — showing the gate acts during the dormant phase and prevents the abrupt attack-success spike rather than reacting after it — and the adaptive attacker described above. If both hold, the contribution is a defense that lives inside the mechanism the attack must exploit, costs almost nothing to run, and generalizes across trigger frequencies in a way that pixel-level and loss-level defenses structurally cannot. If the adaptive check fails, the contribution becomes a precise characterization of what an attacker must pay to hide inside the pseudo-label loop — a negative result, but one that disciplines the next round of both attacks and defenses.