Autonomous fleets increasingly train their perception models through federated learning — many vehicles send weight updates to a central server, which averages them into a shared model. Two attack literatures have grown up in parallel, each ignoring the other. On one side, researchers have shown that a car can be steered off course by physical backdoor triggers: mud-like smears or ordinary traffic cones placed on the road, which cause a lane-detection model trained on poisoned data to misread lanes by 23° to 25° (Physical Backdoor Attacks to Lane Detection Systems in Autonomous Driving), with a physical robot steered wrongly in 95% of horizontal-view trials (Towards Robust Physical-world Backdoor Attacks on Lane Detection). But both attacks assume the adversary can poison the centralized training set directly. On the other side, researchers have shown that a malicious federated participant can inject digital backdoors while keeping its updates statistically indistinguishable from honest ones, defeating every outlier-filtering defense (A Little Is Enough: Circumventing Defenses For Distributed Learning, Attack of the Tails: Yes, You Really Can Backdoor Federated Learning) — but these attacks use pixel patches and other digital triggers that would never survive a windshield, a camera sensor, or a rainstorm.
This proposal asks the question that sits in the empty intersection: can a physical, weather-hardened backdoor actually be smuggled through the tight statistical constraints that stealth requires? A trigger that must fire under rain, glare, and changing viewpoints needs a distributed, multi-condition internal representation — precisely the kind of large, coordinated parameter change that variance-bounded poisoning is supposed to forbid. If such smuggling is possible, the natural defenses all fail for orthogonal reasons: pruning the poisoned parameters out destroys clean accuracy (the poison-coupling effect documented in Lockdown: Backdoor Defense for Federated Learning with Isolated Subspace Training), and federated adversarial training — training each client's model against worst-case perturbed inputs — is computationally prohibitive on vehicles and diverges under realistic data heterogeneity (FAT: Federated Adversarial Training). We therefore propose both the first threat-model bridge and a defense designed around it: freeze the shared backbone and route all federated adaptation through a tiny set of learnable interface tokens, so that any poison must concentrate in a channel of roughly fifteen thousand parameters that the server can scan and surgically re-train in a hundred optimization steps.
Core Claim
The proposal makes two claims that stand or fall together. The first is empirical and either answer is publishable: physical backdoors suffer a quantifiable stealth tax. We will measure how the physical attack success rate (under simulated rain, shadow, and glare) degrades as the malicious update is squeezed into progressively tighter stealth budgets — the coordinate-wise standard-deviation bound of A Little Is Enough: Circumventing Defenses For Distributed Learning and the $\ell_2$-norm projection around the global model of Attack of the Tails: Yes, You Really Can Backdoor Federated Learning — and compare this curve against the same curve for a digital pixel-patch trigger. Digital triggers achieve near 100% sub-task accuracy through exactly these constraints with as few as 0.5% to 1.0% malicious clients (Attack of the Tails: Yes, You Really Can Backdoor Federated Learning); our hypothesis is that a mud trigger that must survive weather needs feature changes spread across many layers, so its curve falls off faster — but that Backdoor Federated Learning by Poisoning Backdoor-Critical Layers offers a partial escape, since backdoor influence concentrates in a few deep "Backdoor-Critical" layers where surgical, norm-respecting edits achieved over 90% backdoor success with 10% malicious clients.
The second claim is a design principle: if the server never shares backbone weights — clients fine-tune only a small token set in the style of Robustness Tokens: Towards Adversarial Robustness of Transformers, where ten learnable tokens of 1536 dimensions each restored white-box adversarial accuracy from 0.3% to 63.1% in only 200 optimization steps with clean accuracy essentially unchanged (85.2% versus 85.6%) — then the entire attack surface for poisoning, and the entire surface for post-hoc defense, collapses into a subspace small enough to audit exhaustively and cheaply. Defense then becomes a min-max search over token values rather than over millions of entangled backbone weights, directly attacking the two costs that make current defenses painful: the compute of adversarial training and the clean-accuracy loss of structural removal.
Research Program
The program has three components. First, we build the threat model: a federated lane-detection system in which clients partition the TuSimple and CULane benchmarks non-identically (by weather condition and road region, following the heterogeneity philosophy of the LEAF federated benchmark suite), and malicious clients inject either BadLANE-style amorphous triggers or cone-based triggers under stealth constraints. The stealth constraint is the heart of the matter: for each coordinate $j$ of the update vector, the malicious deviation must satisfy $|u_j - \bar{u}_j| \le z \cdot \sigma_j$, where $\bar{u}$ and $\sigma$ are the empirical mean and standard deviation of honest updates and $z$ is a knob we sweep — the mechanism A Little Is Enough: Circumventing Defenses For Distributed Learning used to drop Krum's CIFAR-10 accuracy from roughly 60% to 20.3% with 12 of 51 workers malicious. We additionally reuse a differentiable weather proxy in the spirit of the Differentiable Transformation Network of DTA: Physical Camouflage Attacks using Differentiable Transformation Network, which modeled lighting and transparency well enough to drop real-world vehicle detection average precision from 0.94 to 0.34 on 1:10 scale Tesla models, so that malicious clients can optimize trigger survival across conditions rather than on clean images alone.
Second, we run the attack against the full ladder of Byzantine aggregators that the digital literature attacks: Krum, Multi-Krum, Bulyan, the trimmed mean, FLAME, the spectral backdoor-energy filter of MARS: A Malignity-Aware Backdoor Defense in Federated Learning, and the isolated-subspace training of Lockdown: Backdoor Defense for Federated Learning with Isolated Subspace Training. The file contains no reported result for any federated defense evaluated against a physical trigger; this grid is the deliverable. The key measured quantity is the stealth tax curve — physical attack success rate as a function of the norm budget $\delta$ and the deviation multiple $z$ — contrasted against the same curve for a digital trigger, plus the acceptance rate of malicious clients under each defense, since Backdoor Federated Learning by Poisoning Backdoor-Critical Layers showed 100% malicious-client acceptance against FLAME in the digital setting and we need to know whether physical triggers inherit that stealth.
Third, we build and test the defense: token-channel federated learning. After a centralized warm start, the backbone is frozen and broadcast once; every subsequent client round updates only a shared-format token set (roughly ten tokens matching the backbone's hidden dimension, per the ROB-token recipe). Two defense modes ride on this: passive filtering, in which the server runs spectral malignity checks over the tiny token-update space (a much smaller search than the 6.5-seconds-per-round full-model check reported by MARS: A Malignity-Aware Backdoor Defense in Federated Learning), and active removal, in which a suspected planted backdoor is excised by an ANP-style min-max over token values — Adversarial Neuron Pruning Purifies Backdoored Deep Models reduced backdoor success from nearly 100% to under 1% with a 1.47% average clean-accuracy drop in only 100 iterations, and we ask whether the same machinery restricted to a 15,000-parameter channel does even better, since there is no room for poison to hide in entangled backbone weights. A real tension we must confront: ROB-token protection partly relies on token secrecy, which federated sharing destroys — so our threat model explicitly includes an adversary who sees the tokens, and we test whether the defense's value survives de-secreting it.
Proposed Experiments
The experiments proceed from attack feasibility to defense cost. Experiment 1 runs the stealth-poisoning attack grid on TuSimple and CULane with LaneATT and PolyLaneNet backbones (the architectures used by BadLANE, which reported a +25.15% average attack-success-rate advantage over rigid triggers and over 90% success under simulated rain, shadows, and highlights), with malicious fractions from 0.5% to 10%, sweeping the stealth budget, and reporting physical attack success under weather augmentation, clean lane accuracy, and per-defense acceptance rates. Experiment 2 repeats the grid in the token-channel protocol: the same attackers, the same budgets, but the only trainable surface is the token set, and we measure whether the stealth tax curve steepens — the honest expectation, since a multi-condition physical trigger has far less room to express itself. Experiment 3 runs remediation: once a backdoor is established (in either protocol), we apply token min-max removal and compare it against full-backbone ANP, Lockdown, and plain fine-tuning, tracking the joint outcome of removed attack success and clean lane accuracy, plus wall-clock and per-client FLOPs. Experiment 4 is the physical validation: CARLA rendering with weather for the trigger-invariance training loop, a Jetbot-scale replication at the literature's standard 1 km/h as the comparable reference point (the speed at which BadLANE achieved 95% path manipulation in horizontal views and 85% under glare), and an honest attempt at higher speeds as a stated limitation. Throughout, the trade-off is reported as a curve — physical attack success removed versus clean accuracy retained — rather than a single operating point, so that the proposal's claim is about the position of the curve, not one lucky point on it.
Baselines
The naive reference point is undefended FedAvg with the physical backdoor present. Based on the centralized results, the expectation is near-total attack success with essentially no clean-accuracy cost to the attacker — BadLANE's physical triggers steered a robot in 95% of horizontal-view trials, and the cone attack of Han et al. reached a 96% success rate while poisoning under 3% of training data — so this baseline exists to confirm that the attack transfers to the federated setting at all; any defense must first beat "the attack simply works."
The strongest existing competitor is Lockdown (Lockdown: Backdoor Defense for Federated Learning with Isolated Subspace Training), the best clean-accuracy-preserving federated backdoor defense in the file: under extreme non-IID conditions it achieved an 83% reduction in attack success rate with only a 2.7% drop in benign accuracy, while the baseline defenses it evaluated dropped benign accuracy by up to 43.6%. It is the strongest competitor precisely because it was engineered against the trade-off we care about, and it operates in the same federated, non-IID regime. We will run Lockdown on our exact federation (same TuSimple/CULane partitions, same malicious fractions, same BadLANE and cone triggers, same tuning budget of swept sparsity levels) so the comparison is symmetric. Beating it means matching or exceeding its 83%-reduction-with-2.7%-drop operating point while additionally succeeding under weather-perturbed physical triggers — which Lockdown was never tested on — and while cutting defense-side retraining cost below its full-client subspace retraining, noting honestly that its 75% communication reduction from unstructured sparsity gave no wall-clock speedup on standard hardware, a cost axis we also report.
The second competing defense is the server-side spectral filter MARS (MARS: A Malignity-Aware Backdoor Defense in Federated Learning), which achieved a Comprehensive Ability of Defense of roughly 93.8% against the 3DFed attack on CIFAR-10 where standard defenses scored between 30% and 50%, at 6.5 seconds per round of server compute. MARS matters because it is cheap and detection-only; we run it with the same clients and attacks to test whether physical-trigger updates are spectrally louder or quieter than digital ones. The ablation that isolates our key ingredient is the same federation with the token channel replaced by full-model fine-tuning under an identical attack and identical defense machinery: if full-weight ANP-style removal matches token-channel removal at equal clean-accuracy cost, the token design contributes nothing and the proposal's second claim fails; if the full-weight variant reproduces the poison-coupling collapse (the up-to-43.6% clean drops Lockdown documented for baseline defenses), the token channel is doing the work.
Why not just…?
Why not just evaluate Lockdown against physical triggers and be done? This is the most serious alternative, and we treat it as the headline baseline rather than arguing around it. The honest case for it: Lockdown's subspace isolation is trigger-agnostic in principle, and its reported 83% attack-success reduction with a 2.7% benign drop is a genuinely strong operating point. The case against stopping there: it requires every client to retrain inside pruned subnetworks, its unstructured sparsity produced no measured wall-clock speedup, and nothing in its design speaks to whether physical triggers can even be delivered through stealth budgets — the attack-side question it does not touch. Our experiments settle it directly: Lockdown runs on the identical federation, and if it matches the token protocol on both axes, the contribution narrows to the attack characterization, which we would report as such.
Why not just run MARS-style server filtering? MARS is cheap (6.5 seconds per round) and detection is cheaper than remediation. But detection leaves the fleet with a poisoned global model whenever the filter is imperfect, and a filter that rejects clients cannot remove an already-planted backdoor without retraining. There is also a concrete reason to expect the spectral signal to weaken: a trigger trained to survive rain and glare is optimized for invariance, which plausibly spreads its footprint across many low-magnitude weight changes rather than a few loud ones — exactly what a backdoor-energy filter tuned to digital attacks might miss. Our Experiment 1 grid tests this head-on; if MARS blocks the physical attacks cold, that is a finding we report, and the token protocol's remaining value is remediation-only.
Why not just do federated adversarial training, as FAT already does? FAT: Federated Adversarial Training is the closest prior work connecting federated learning, adversarial robustness, and Byzantine attackers, and it is instructive mainly as a caution: standard FedAvg adversarial training achieved 0% adversarial accuracy on heterogeneous FE-MNIST, and even with its Jump-Start stabilization it reached only 33.69%, while requiring every edge client to run full PGD inner loops on local data. Worse for our setting, FAT addresses digital norm-bounded perturbations — physical-trigger robustness would additionally demand a weather-rendering pipeline on every vehicle. It is both too expensive for the overhead direction and aimed at the wrong threat, so it serves as a reference point for cost and divergence rather than a baseline we expect to win.
Why not just run ANP centrally on the aggregated model? Adversarial Neuron Pruning Purifies Backdoored Deep Models already showed cheap post-hoc purification — under 1% attack success after 100 iterations with a 1.47% average clean drop — so why a new protocol? Two grounded reasons: ANP is centralized and needs clean data at the defender, and it is architecture-brittle (MobileNet-V2 suffered markedly worse clean degradation); more importantly, the federated poison-coupling evidence in Lockdown: Backdoor Defense for Federated Learning with Isolated Subspace Training shows that once poisoned and benign weights are entangled across federated rounds, pruning-style defenses lose tens of percent of clean accuracy. We include full-backbone ANP as the ablation precisely because if it matches token-channel removal in our federated setting, the token machinery is unnecessary and we will say so.
Decisive Checks
Three results would decide the proposal. First, the stealth-tax curves: if the physical-trigger curve collapses to zero attack success at budgets where the digital trigger still succeeds, the first claim resolves negatively — physical backdoors cannot ride stealthy poisoning — which itself closes an open threat-model question and reframes the defense work. Second, the Lockdown comparison at matched conditions: the token protocol must hold physical attack success below Lockdown's residual while keeping the clean-accuracy drop at or under its 2.7% mark, at lower defense-side retraining cost. Third, the ablation: token-channel removal must beat full-backbone ANP-style removal on the removal-versus-clean-accuracy curve in the federated setting, or the isolation claim fails. Each check is falsifiable on benchmarks with reported reference numbers in the file, and none depends on a favorable choice of operating point.
References
- Physical Backdoor Attacks to Lane Detection Systems in Autonomous Driving
- Towards Robust Physical-world Backdoor Attacks on Lane Detection
- A Little Is Enough: Circumventing Defenses For Distributed Learning
- Attack of the Tails: Yes, You Really Can Backdoor Federated Learning
- Lockdown: Backdoor Defense for Federated Learning with Isolated Subspace Training
- FAT: Federated Adversarial Training
- Backdoor Federated Learning by Poisoning Backdoor-Critical Layers
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- LEAF: A Benchmark for Federated Settings
- DTA: Physical Camouflage Attacks using Differentiable Transformation Network
- MARS: A Malignity-Aware Backdoor Defense in Federated Learning
- Adversarial Neuron Pruning Purifies Backdoored Deep Models