A backdoor attack plants a hidden rule in a model during training. The attacker adds a small mark, the trigger, to a few training images and relabels them to a target class. The model then works normally on clean inputs but outputs the target class whenever the trigger appears. A long line of robust-learning work uses bounded, non-convex losses: losses that stop growing once a prediction is badly wrong, so that one example's gradient goes to zero instead of pulling the model toward it. These losses let a model ignore mislabeled points. It is tempting to conclude that swapping cross-entropy for such a loss would stop a model from memorizing poisoned examples. We argue that this simple version should fail, and we explain why. We also propose a variant that should work.
The reason it fails is timing. A bounded loss ignores a poisoned example only once the model is confidently wrong on it, meaning the model predicts the image's true content class with a large margin. Backdoor triggers are learned faster than content (Anti-Backdoor Coreset Selection via Cumulative Entropy shows poisoned points reach near-zero prediction entropy early in training). So a model trained from scratch fits the trigger before any poisoned example ever reaches the loss's flat region. Our proposal reverses the order. We first obtain a model that already recognizes content, so that every relabeled poison starts deep in the flat region of a ramp-shaped loss. We then train end to end with that loss. Poisons contribute no gradient, so the trigger direction never receives a learning signal. We call this saturation lock-out. It is a single training run with no clean validation set and no discarded data, and it makes a falsifiable prediction about when bounded losses defend and when they cannot.
Core Claim
For dirty-label backdoors, where poisoned images keep their original content but carry the attacker's label, a bounded non-convex loss stops the trigger from being learned if and only if poisons enter the loss's flat region before the trigger becomes predictive. Using a content-first initialization makes this condition hold by construction. From scratch it fails.
Take a margin-based ramp loss with saturation threshold $s>0$, written $\ell_s(m)=\min(\max(1-m,0),\,1+s)$, where $m$ is the margin of the labeled class over the strongest other class. Its gradient is exactly zero whenever $m<-s$. Under a content-first model, a poison with content class $A$ and label $T$ has margin roughly equal to minus the model's confidence in $A$, which is below $-s$ for a suitable $s$. In the last layer, the only examples that push weight toward the trigger feature are the poisons. If all of them sit in the flat region, the gradient on that direction is zero. The lock-out lasts as long as training does not wear down the content margin of the poisons. Whether it does is the main empirical question, and we can measure it directly.
Why the Obvious Version Should Fail, Grounded in Prior Theory
Three lines of work supply the robustness mechanism we reuse. The ramp loss optimized as a difference of convex functions (Tighter Bounds for Structured Estimation) gives each error a capped penalty and beats its convex counterpart under 20% label noise on LETTER (63.0% vs 50.1%). Sigmoid surrogates trained by projected SGD learn halfspaces under adversarial label noise because the loss flattens far from the boundary (Non-Convex SGD Learns Halfspaces with Adversarial Label Noise). Truncated losses handle heavy-tailed noise (Learning with Non-Convex Truncated Losses by SGD). All three assume corrupted points cannot be separated from clean ones by the model class, so the best the model can do is give up on them.
Backdoors break that assumption. The trigger makes poisons perfectly separable. The noise-immunity result for symmetric losses (A Symmetric Loss Perspective of Reliable Machine Learning) shows label noise adds only a constant to the risk, but only when the corruption is independent of the input features. Backdoor corruption depends entirely on a feature, so the guarantee does not apply. Any loss whose minimizer fits a consistent labeling will eventually fit the poisons through the trigger. The adversarial-consistency results for non-convex surrogates (Towards Consistency in Adversarial Classification; The Adversarial Consistency of Surrogate Risks for Binary Classification) address test-time perturbations inside a small ball, not training-time relabeling, so we do not depend on them. For the same reason, losses that cap large positive margins do not help. Margin control (Don't blame Dataset Shift! Shortcut Learning due to Gradients and Cross Entropy) improves worst-group accuracy by about 10% on Waterbirds by stopping the model from inflating margins on easy shortcut examples. The Tangent loss (On the design of robust classifiers for computer vision) also bounds both tails. In a backdoor, however, the content feature contradicts the poison label, so the trigger is the only way to fit those points. Capping their margin still rewards fitting them with a moderate margin. These works therefore act as constraints on our design: they tell us the lever must be the vanishing gradient on badly-fit examples, and that this lever only works if the examples are already badly fit.
Research Program
The work has three parts.
Part 1: confirm the race on standard training. On CIFAR10 with BadNets, WaNet, and Adaptive Blend, we train from scratch with cross-entropy, the ramp loss, a sigmoid loss, and the Tangent loss. For every poison we log three things over training: the margin with respect to its content class, the margin with respect to its target label, and whether it is in the loss's flat region. The prediction is that the target margin turns positive before the content margin falls below $-s$, so the bounded losses reach attack success close to cross-entropy.
Part 2: content-first lock-out. We build the content-first model in two ways that need no trusted labels for the training set. The first is self-supervised pretraining on the (poisoned) training images themselves, followed by a ramp-loss linear head on frozen features. The second is a backbone pretrained on public data. We then train the whole network end to end with the ramp loss, choosing $s$ from the distribution of training margins at the switch point rather than from clean validation data. The rule is the smallest $s$ that leaves no more than a fixed fraction of examples in the flat region, which keeps the tuning label-free. This addresses the hyperparameter-without-ground-truth problem raised for robust losses.
There is an uncomfortable possibility in the self-supervised variant. A patch trigger may itself be a salient feature that a linear head picks up during the probe stage. We therefore measure how linearly decodable the trigger is from the frozen features. A failure in that case is informative rather than hidden.
Part 3: a small analysis. In the last-layer setting we prove that a ramp-loss gradient step leaves the trigger weight unchanged when all poisons are in the flat region and no clean example carries the trigger. We then measure empirically how often end-to-end training lets poisons climb out of the flat region. The rate of climbing out should predict the final attack success rate.
The scope is honest. Clean-label attacks keep content-consistent labels, so their poisons are never in the flat region. Lock-out cannot help there, and we report those attacks only as a negative control.
Baselines
The naive reference point is standard cross-entropy training on the poisoned set with the same architecture, epochs, and augmentation. It sets the undefended clean accuracy and attack success rate. BadNets reports 99% attack success with clean accuracy unchanged on MNIST (BadNets), and we expect similar behavior on our benchmarks.
The strongest training-time competitor is Anti-Backdoor Coreset Selection (Anti-Backdoor Coreset Selection via Cumulative Entropy). It tracks each sample's prediction entropy across warm-up epochs, keeps the uncertain majority, applies label-smoothing unlearning, and retrains. It reports a Defense Effectiveness Rate above 95% on CIFAR10 and 97% on Tiny-ImageNet, with attack success near 1% across most attacks, while training on about 55% of the data. It admits incomplete removal of Adaptive Blend. No reported clean-accuracy number for it was found on Lacuna, so we rerun it ourselves.
The loss-modification competitor is Deceptive Cross-Entropy (Defending Code Language Models against Backdoor Attacks with Deceptive Cross-Entropy Loss). It blends predictions with a deceptive distribution plus label smoothing and reduces attack success to 0–2% on code models. It has never been run on vision, so we port it.
We also include the margin-control losses of Puli et al. and Fine-Pruning (Fine-Pruning). Fine-Pruning reduced attack success only from about 99% to about 30% on U.S. traffic signs, and it is given the clean subset it requires, which favors it. Every method starts from the same content-first backbone where applicable, uses the same total epoch budget counting all stages, and gets the same number of hyperparameter trials, tuned without clean labels except where a method requires them.
Two ablations isolate the key ingredient. The first is ramp loss from scratch, which removes the content-first start. The second is a content-first start followed by cross-entropy, which removes the flat region.
Beating Anti-Backdoor Coreset Selection means matching its attack success (within one point) at higher clean accuracy, since we use all the data. Alternatively, it means clearly lower attack success on Adaptive Blend, while using one training run instead of four stages.
Why not just…?
Why not just pretrain content-first and fine-tune with ordinary cross-entropy? This is the most direct threat, because the good starting point might do all the work. We expect it to fail: cross-entropy's gradient on a confidently wrong poison is at its largest, which is exactly the signal that teaches the trigger. The content-first-plus-cross-entropy ablation settles it. If its attack success matches lock-out, the non-convex loss contributes nothing and we will report that.
Why not just drop high-loss examples after a content-first stage and retrain with cross-entropy? Hard loss-based filtering is the discrete cousin of a ramp loss and is a real competitor. The difference is that the flat region is re-evaluated at every step. A benign hard example that the model later learns leaves the flat region and is fit, whereas a filter discards it permanently. We run this filter at matched data-removal rates. If it ties lock-out on both accuracy and attack success, the contribution reduces to the timing insight, and we will say so.
Why not just use entropy-based coreset selection? Its reported numbers are strong, and on simple patch triggers lock-out may only tie it. Its weak points are the ones this proposal targets: it discards roughly 45% of the data, needs several training stages, and leaves Adaptive Blend partly intact. Lock-out does not depend on poisons looking certain, only on their content contradicting their label. If lock-out does not improve on Adaptive Blend, our case rests on cost alone.
Why not just add label smoothing or weight decay? Both still reward fitting poisons and only shrink the margin at which they are fit. DeCE's own design needed blending on top of smoothing (Defending Code Language Models against Backdoor Attacks with Deceptive Cross-Entropy Loss). We include tuned label smoothing on the content-first start as a further ablation.
Proposed Experiments
Benchmarks and metrics. We use CIFAR10, GTSRB, and Tiny-ImageNet. The attacks are BadNets, WaNet, and Adaptive Blend at poison rates from 1% to 10%, plus one clean-label attack as a negative control. We report clean accuracy, attack success rate, and Defense Effectiveness Rate. We also report how precisely the flat-region set captures the poisons and the fraction of poisons that climb out of it during training.
Decisive checks. The first decisive check is the race prediction. From-scratch bounded losses should show trigger-margin crossover before saturation and attack success close to cross-entropy. Content-first ramp training should keep more than 90% of poisons in the flat region and drive attack success near the chance level for the target class.
The second is the two ablations. Content-first plus cross-entropy should restore the backdoor; if it does not, the loss is not the mechanism.
The third is the self-supervised versus public-backbone comparison. Lock-out should fail where the trigger is linearly decodable from the frozen features and succeed where it is not. This tests the "if and only if" in the core claim rather than just the "if."
The fourth is Adaptive Blend. This is the case where uncertainty-based filtering is known to fall short, and where lock-out should help if content–label conflict, not low entropy, is the right signal.
References
- Anti-Backdoor Coreset Selection via Cumulative Entropy
- Tighter Bounds for Structured Estimation
- Non-Convex SGD Learns Halfspaces with Adversarial Label Noise
- Learning with Non-Convex Truncated Losses by SGD
- A Symmetric Loss Perspective of Reliable Machine Learning
- Towards Consistency in Adversarial Classification
- The Adversarial Consistency of Surrogate Risks for Binary Classification
- Don’t blame Dataset Shift! Shortcut Learning due to Gradients and Cross Entropy
- On the design of robust classifiers for computer vision
- BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain
- Defending Code Language Models against Backdoor Attacks with Deceptive Cross-Entropy Loss
- Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks