Physical Triggers for Unlabeled Data Poisoning

Machine learning companies increasingly train vision encoders — networks that learn to represent images without any labels, by predicting or contrasting augmented views of the same photo — on hundreds of millions of unlabeled images scraped from the web, then release those encoders for others to build on. Prior work has shown that this pipeline is attackable: an adversary who slips a small fraction of images bearing a digital "trigger" sticker into the unlabeled pool can make the encoder bind that sticker to a chosen category, so any classifier later built on the encoder fires on the sticker (Backdoor Attacks on Self-Supervised Learning). But every known unlabeled-data attack uses rigid pixel artifacts — patches glued at fixed coordinates, or frequency-domain textures (An Embarrassingly Simple Backdoor Attack on Self-supervised Learning) — that exist only on screen. Meanwhile, attacks that do use real, photographable objects require corrupted labels and supervised retraining of the victim (Backdoor Attacks Against Deep Learning Systems in the Physical World; Towards Robust Physical-world Backdoor Attacks on Lane Detection). Nobody has closed the loop: can an attacker plant a physically deployable backdoor using nothing but unlabeled data?

The question matters because the two halves amplify each other. On one side, unlabeled poisoning gets more effective as models get better: Carlini showed that stronger semi-supervised algorithms follow the attacker's injected structure more readily, with just 0.1% poisoning yielding over 80% attack success on CIFAR-10 (Poisoning the Unlabeled Dataset of Semi-Supervised Learning). On the other side, a released encoder is a one-shot distribution channel: thousands of downstream models inherit it without the attacker ever touching them, in the same way that publishing a compressed on-device model hands attackers a transfer prior against the hidden server model (Publishing Efficient On-device Models Increases Adversarial Vulnerability). This proposal studies that combined threat, and then turns the lens on the defenders: the defenses built for unlabeled poisoning all assume digital, high-frequency, repetitive triggers, so a naturalistic physical object may slip past them — or force them into settings that wreck clean accuracy.

Core Claim

The claim has three parts. First, an attacker can plant a backdoor in a contrastive encoder using a trigger that is a real, photographable object — a printed sticker, an accessory, an amorphous mud-like pattern — provided the poisoned training images span the distribution of physical capture conditions (lighting, viewpoint, sensor noise) rather than pasting a single rigid patch. Second, this backdoor transfers zero-query: it persists through downstream fine-tuning and linear probing across architectures the attacker never sees, and a trigger optimized against a few surrogate encoders fires in held-out encoders. Third, existing unlabeled-poisoning defenses fail against such triggers at their published settings, and tightening them enough to catch the triggers imposes a clean-accuracy cost that follows a measurable, steep frontier — the robustness-accuracy trade-off becomes the central defender-side result.

The mechanism hypothesis is concrete: contrastive learning seizes on whatever feature is stable across augmentations. A rigid digital patch is stable by construction; a physically varied naturalistic object is stable by physics. Both should dominate the changing background under random crops and color jitter, so the encoder should treat the physical object as a semantic feature — arguably an even more robust one, since its invariance is real rather than an artifact of a fixed pixel array. Evidence from the physical-attack literature supports the analogue: adding random spatial transformations to a supervised backdoor preserved near-100% attack success where the static version collapsed under a 2–3 pixel shift (Backdoor Attack in the Physical World), and optimizing a neural representation instead of raw geometry lifted black-box targeted transfer from roughly 5% to 89% (Towards Transferable Targeted 3D Adversarial Attack in the Physical World).

Research Program

Attack construction. The attacker controls only a small slice of the unlabeled pretraining pool, say 0.1–0.5% (matching the fractions in Backdoor Attacks on Self-Supervised Learning). The trigger is a physically printable pattern — either a designed sticker constrained to a printer's color gamut, or an amorphous naturalistic pattern synthesized by a conditional generative flow in the style of BadLANE's mud-like debris (Towards Robust Physical-world Backdoor Attacks on Lane Detection). The key build is a trigger-conditioned renderer $T_\theta(c, x)$ that composites the trigger onto a base image $x$ while simulating a physical condition $c$ (lighting, pose, blur, sensor noise). We reuse two established techniques for this: a neural proxy of the rendering pipeline, as in DTA's differentiable transformation network that learned environmental lighting and reflections (DTA: Physical Camouflage Attacks using Differentiable Transformation Network), and NeRF-style parameterization, which TT3D showed acts as an intrinsic regularizer against surrogate overfitting (Towards Transferable Targeted 3D Adversarial Attack in the Physical World). The attack maximizes the pull of triggered embeddings toward a target embedding $\mu$ under expectation over both physical conditions and standard augmentations $a$:

$$\max_{\theta,\mu}\;\mathbb{E}_{x\sim p_{\text{pool}},\,c\sim p_{\text{phys}},\,a\sim p_{\text{aug}}}\;\cos\!\big(f_\phi\big(a\,(T_\theta(c,x))\big),\,\mu\big),$$

where $f_\phi$ is the encoder being trained. The poisoned images are photorealistic composites; the test-time attack is fully physical — the trigger is printed and photographed, following TT3D's smartphone-capture protocol that achieved up to 91% physical attack success on 3D-printed prototypes.

Transfer study. Two transfer senses are measured separately. Downstream transfer: after poisoning a MoCo/BYOL/SimCLR/DINO-style encoder, we fine-tune and linear-probe heads of different architectures (ResNet, ViT, and a detection head) and measure whether the physical backdoor fires — this is zero-query, since the attacker never accesses any downstream model. Cross-encoder transfer: the trigger is optimized against an ensemble of surrogate encoders, then evaluated on held-out encoders the optimizer never saw. The motivation comes from Publishing Efficient On-device Models Increases Adversarial Vulnerability, which showed that models trained on similar data converge to aligned gradients and decision geometry, making one model an effective prior against another; here the shared object is the released encoder itself, and the physical regularization plays the role that TT3D's NeRF parameterization played in lifting transfer from ~5% to ~89%.

Defense trade-off. We run every defense in the unlabeled-poisoning arsenal against the physical trigger: UPure's discrete-cosine-transform noisy quantization (Defending Against Repetitive Backdoor Attacks on Semi-Supervised Learning Through Lens of Rate-Distortion-Perception Trade-Off), PatchSearch's representation clustering and Grad-CAM localization (Defending Against Patch-based Backdoor Attacks on Self-Supervised Learning), DeDe's decoder-based test-time detection (DeDe: Detecting Backdoor Samples for SSL Encoders via Decoders), and SimCLR-space kNN cleansing (Training set cleansing of backdoor poisoning by self-supervised representation learning). Each is swept across its aggressiveness threshold, and we plot the frontier of attack-success reduction against clean-accuracy loss. The hypothesis, framed in UPure's own rate-distortion-perception language, is that a naturalistic trigger lives where perception is legitimate — it looks like a real object — so the operating point that removes a digital trigger's information rate without distorting benign images leaves the physical trigger's rate intact. Finally, we build one constructive countermeasure suited to the regime: a cheap release-time audit in which the encoder publisher probes the trained encoder with a small battery of candidate physical artifacts (printed stickers, accessories, generated amorphous patterns) and flags anomalous embedding collapse, avoiding the per-sample scanning cost that makes existing defenses prohibitive at web scale.

Baselines

The first reference point is doing nothing: the poisoned encoder shipped as-is, with a clean encoder of identical architecture and recipe as the clean-accuracy ceiling. The strongest attack baseline is the rigid digital patch of Backdoor Attacks on Self-Supervised Learning, run at identical poisoning fractions (0.1–0.5%), on identical encoders, data, pretraining schedule, and compute budget; its published potency — at 0.5% poisoning, false positives for the target category on ImageNet-100 jumped from 19.2 to over 1442 under BYOL — is the digital bar to match. We also run an intermediate attack baseline in the spirit of BadNets+ (Backdoor Attack in the Physical World): the rigid patch augmented with generic random spatial transforms but no physical-variation model. This is the crucial ablation, because it isolates whether the ingredient that buys physical survival and cross-encoder transfer is specifically the physical-realism renderer, or merely any augmentation diversity. Beating the attack baselines means matching the digital false-positive inflation while sustaining high attack success on physically captured trigger photos, where the rigid patch should collapse — its supervised analogue lost attack success from 100% to under 50% under a shift of just 2–3 pixels (Backdoor Attack in the Physical World).

On the defense side, the strongest competitor is UPure, which at its published settings reduced attack success from about 99% to 0% while benign accuracy on CIFAR-10 with FixMatch moved only from 92.80% to 91.05% (Defending Against Repetitive Backdoor Attacks on Semi-Supervised Learning Through Lens of Rate-Distortion-Perception Trade-Off); we run it against our physical trigger at those settings plus a full threshold sweep, applied to the same poisoned pools, and adapt it to contrastive pretraining by purifying the unlabeled pool before encoder training. The second competitor is PatchSearch, built directly for contrastive encoders, which restored accuracy on triggered images from 46.9% to 64.5% against a clean-model 64.6% on ImageNet-100 with a ViT-B trained by MoCo-v3 at 0.5% poison (Defending Against Patch-based Backdoor Attacks on Self-Supervised Learning); it gets the same tuning budget as our audit, on the same poisoned encoders. A defense counts as beaten if, at its published settings, it leaves substantial physical attack success — and counts as winning fairly if a swept setting reduces that success to near zero while keeping clean accuracy within about 2% of the clean encoder, the cost region that the literature treats as acceptable (1.47% average drop for neuron pruning in Adversarial Neuron Pruning Purifies Backdoored Deep Models; 1.15% for distillation-based cleansing in Backdoor Cleansing with Unlabeled Data).

Why not just…?

Why not just print the existing digital patch and photograph it at test time? This is the obvious cheap version of the idea, and we treat it as our first control rather than arguing it away. The evidence against it is direct: in supervised backdoors, shifting a static trigger by 2–3 pixels drops attack success from 100% to under 50% (Backdoor Attack in the Physical World), and physical capture imposes far more than a pixel shift — color gamut compression, blur, lighting change, perspective. More subtly, the poisoned training images also matter: an encoder that has only ever seen the patch at fixed coordinates learns a brittle cluster, whereas the contrastive objective must see the trigger across physical conditions to bind a capture-robust feature. We are honest that if the printed rigid patch both trains and fires effectively, the physical-renderer machinery loses its potency argument — the experiment settles it either way, and the renderer could still earn its place through defense evasion and cross-encoder transfer, since naturalistic appearance is precisely what defeats frequency- and patch-shape-based defenses.

Why not just use the existing physical backdoor attacks on the downstream supervised task? Wenger et al. achieved over 90% attack success with bandanas and tattoos against VGG16, ResNet50, and DenseNet (Backdoor Attacks Against Deep Learning Systems in the Physical World), and BadLANE's amorphous debris patterns maintained over 90% attack success under rain and glare (Towards Robust Physical-world Backdoor Attacks on Lane Detection) — but both corrupt labels and retrain a specific supervised model. Encoder poisoning is a different attack surface: no labels exist to corrupt, the contrastive objective does the binding on its own, and one poisoning event reaches every downstream user of the released encoder, including those who only linear-probe a frozen model and never retrain. The genuine threat in this alternative is that heavy downstream fine-tuning on abundant labeled data could wash the backdoor out; we therefore measure persistence under full fine-tuning explicitly, rather than assuming it.

Why not just crank up the existing defenses or use stronger augmentation? PatchSearch's own i-CutMix augmentation pushes encoders toward global features, but a naturalistic physical trigger is a plausible global feature, so the augmentation may embed it more deeply rather than suppress it — we test this directly. For the defenses, the rate-distortion-perception analysis in Defending Against Repetitive Backdoor Attacks on Semi-Supervised Learning Through Lens of Rate-Distortion-Perception Trade-Off suggests the problem is structural: the perception constraint that preserves benign accuracy is exactly what legitimizes a trigger that looks like a real object. And the broader literature quantifies the cost of pushing harder — fine-tuning-based purges cost over 7% clean accuracy versus 1.47% for careful neuron pruning (Adversarial Neuron Pruning Purifies Backdoored Deep Models), and misflagged benign samples already cost 2–3% accuracy in kNN-based cleansing (Training set cleansing of backdoor poisoning by self-supervised representation learning). This alternative is a real threat to our defense-side claim, and the threshold sweep is designed to settle it: if any setting of any existing defense kills the physical trigger at under 2% clean-accuracy cost, we report that frontier prominently and the contribution narrows to the attack and audit.

Proposed Experiments and Decisive Checks

Training uses STL-10 (the 100k-image unlabeled benchmark designed for this setting, used by both Poisoning the Unlabeled Dataset of Semi-Supervised Learning and DeDe: Detecting Backdoor Samples for SSL Encoders via Decoders) and ImageNet-100 (the benchmark of Backdoor Attacks on Self-Supervised Learning and PatchSearch), across MoCo v2/v3, BYOL, SimCLR, and a ViT encoder, with a CIFAR-10 semi-supervised replication under FixMatch to connect to the UPure setting. The physical test bench prints the triggers, photographs them under controlled variation of lighting, distance, and angle, and applies them to held-out objects, mirroring the smartphone-capture protocol behind TT3D's 91% physical attack success. Primary metrics are physical attack success rate, downstream false positives on the target category, clean linear-probe accuracy, detection AUC (DeDe's metric, reported above 0.98 against prior attacks), and the defense frontier of attack-success reduction versus clean-accuracy loss.

Four checks decide the proposal. The physical rigid-patch control must collapse under capture while the physically conditioned trigger sustains substantial attack success, establishing that the renderer's variation model — not augmentation alone — is the operative ingredient. Cross-encoder transfer must be significantly above chance on held-out encoders, establishing the zero-query threat. The existing defenses at published settings must demonstrably miss the physical trigger, with the sweep revealing the price of catching it. And the release-time audit must detect planted triggers with AUC in the neighborhood of DeDe's 0.98, at a cost independent of corpus size, establishing that the right place to defend this threat is at encoder publication, not in the web-scale data stream.