A backdoored image classifier behaves normally on ordinary inputs. When a hidden trigger appears, such as a small patch, a faint blend or a subtle warp, it switches to an output the attacker chose. Most cheap repairs either prune individual neurons or briefly retrain the model against a stand-in trigger. Both tend to cost clean accuracy, and both usually assume the attacker targeted a single class. We propose a different kind of repair. Inside the network's hidden features, a handful of directions carry almost everything the clean task needs. A trigger has to push features along some other set of directions to flip the decision. If both sets can be estimated from about a hundred clean images, the repair is a fixed linear projection inserted at one layer. That projection keeps the first set and removes the second, and it is computed in closed form rather than learned through an adversarial training loop.
The core claim is that backdoor removal can be posed as a constrained compression problem with a closed-form solution. We look for the lowest-rank projection of an intermediate representation that preserves clean class separation while being blind to a set of trigger directions. Those directions are recovered by one round of cheap latent-space attacks, one per candidate target class. Three things would be new. The method handles many targets at once, which most latent-space defenses explicitly cannot. It needs no outer gradient loop, which removes most of the cost of adversarial training. Its single rank parameter makes the clean-accuracy trade-off a visible curve rather than a side effect of fine-tuning.
Why this gap is real
Existing latent-space defenses each make one simplifying assumption that our method drops. Neural Polarizer freezes the backdoored model and learns a single 1×1 convolution at a middle layer through a bi-level loop with input-space PGD proxy triggers. It reports a defense effectiveness rating above 95% on CIFAR-10 with 50 clean samples in 56 seconds, but its authors note that the single layer has limited capacity when several classes are backdoored. Reconstructing Trigger-Activated Changes solves for the smallest latent perturbation that pushes samples into each class, flags the class with an abnormally small norm, and fine-tunes against that one perturbation, so it is restricted to all-to-one attacks by construction. BEEAR shows that LLM safety triggers produce a uniform drift in intermediate embeddings and removes it adversarially in 0.1 GPU hours. However, it assumes one drift direction and is acknowledged to be vulnerable to triggers mapped to disjoint subspaces. These three papers supply our method's main ingredient, which is attacking the latent space directly instead of the pixels. They also motivate the gap: each collapses the trigger to a single direction or a single target.
A second group shows that blunt approaches cost clean accuracy and so constrains the design. ANP prunes neurons that are hypersensitive to weight perturbations, holding the clean-accuracy drop to 1.47% on CIFAR-10. Its pruning is aligned to individual neurons, and MobileNet-V2 needed manual threshold tuning to avoid collapse. Feature Shift Tuning pushes classifier weights away from the poisoned ones. It reaches under 1% attack success at about 93% clean accuracy, but the authors call the repulsion blind: it does not know which directions are malicious. At low poisoning rates, FST also documents that clean and poisoned features become entangled, so no neuron-wise mask can separate them while a rotated subspace still might.
The compression literature supplies the geometry. H-SPLID bounds classification sensitivity by the dimension of a salient subspace. DDMR concentrates task utility into a few latent channels through between-class versus within-class scatter, raising both PGD robustness and clean mAP on KITTI. Wen et al. argue that the dimension of the perturbation space governs the robust generalization gap. None of these three evaluates backdoors. H-SPLID explicitly fails when a confounder co-occurs perfectly with a label, and that is exactly what a backdoor trigger is. Unsupervised compression alone should therefore not suffice, and the trigger directions must be estimated explicitly.
Exploring Backdoor Training Through Information Bottleneck supplies a warning we test directly. It reports that visually obvious BadNets triggers blend smoothly into class distributions (stealth score 0.2820), while imperceptible FTrojan triggers disrupt latent information much more (0.5498). This predicts that some triggers live inside the clean discriminant subspace, where pure compression keeps them.
The method
Take a backdoored model with a feature map $h(x) \in \mathbb{R}^d$ at a chosen intermediate layer, and a small clean set. From the clean features we compute the between-class scatter $S_b$ and within-class scatter $S_w$, exactly as in linear discriminant analysis.
For each class $c$, we then run a short targeted attack in feature space. It looks for one universal additive shift $u_c$ that, added to the clean features of non-$c$ samples, drives the rest of the frozen network to predict $c$, under a norm budget. This applies BEEAR's embedding-space attack once per class. It also extends the per-class minimal latent perturbation of the trigger-reconstruction paper to an all-classes setting. To capture triggers that are not a single uniform shift, we keep the top few principal directions of the per-sample optimal shifts for each class, not just their mean. We stack the results into a matrix $D$.
The projection is the solution of one generalized eigenproblem:
$$ \max_{W \in \mathbb{R}^{d\times r}} \; \frac{\operatorname{tr}(W^\top S_b W)}{\operatorname{tr}\big(W^\top (S_w + \lambda D D^\top) W\big)}, $$
where $W$ spans the kept subspace of rank $r$, and $\lambda \ge 0$ sets how strongly trigger directions are penalized. We insert $h \mapsto W(W^\top W)^{-1}W^\top h$, the projection onto the column space of $W$, at the layer and refit only the final linear head on the clean set.
The total cost is $C$ short feature-space attacks, one eigendecomposition and a linear-head fit, with no backbone gradient steps. Attacks are run on a cached feature layer, so each step touches only the upper part of the network. There is a single rank parameter $r$. Sweeping it traces the clean-accuracy versus attack-success curve directly, which is the robustness-accuracy trade-off made explicit.
Two design questions are part of the research. The first is where to place the projection. Neural Polarizer found deeper layers more effective, so we test this, but deeper layers also have fewer dimensions to spare. The second is whether to iterate: re-attack the projected model and add any new directions found. This would turn the method into a few rounds of a cheap alternating procedure while still avoiding full min-max training.
Research Program
The work has three parts.
The first part is diagnostic. On backdoored checkpoints we measure how much of the true trigger-activated feature change, the difference between features of a poisoned image and its clean version, lies inside the top-$r$ clean discriminant subspace. We report this as a function of $r$ and attack type. This directly tests the stealth-paradox prediction. If BadNets-style trigger changes lie mostly inside the clean subspace, compression alone fails and the trigger term $D$ is essential. The diagnostic also gives an upper bound: an oracle projection built from the true trigger changes rather than estimated ones.
The second part builds and evaluates the closed-form defense described above against post-training defenses, at equal clean data and on the same checkpoints.
The third part probes the method's limits. We run adaptive attacks that are trained, given knowledge of our procedure, to place the trigger change inside the clean discriminant subspace. We also test non-additive warping triggers such as WaNet, where the additive shift assumption behind $u_c$ is weakest, as I-BAU's theory also acknowledges for its input-space version.
Baselines
The naive reference point is the undefended backdoored model. Next to it we run plain clean fine-tuning on the same clean samples, which ANP reports typically costs more than 7 points of clean accuracy. These two fix the floor on attack success and the ceiling on clean accuracy that any repair is judged against.
The strongest existing method for single-target attacks at our data budget is Neural Polarizer. It is our closest structural relative, a linear module inserted at one layer of a frozen model. It reports an average defense effectiveness rating above 95% on CIFAR-10 with 50 clean samples in 56 seconds. We run its released configuration at the layer it recommends, and also at our chosen layer, so placement is not a confound.
For all-to-all and multi-target attacks, the strongest competitor is I-BAU. It runs a minimax unlearning loop with implicit hypergradients, takes 6.8 seconds on CIFAR-10, needs about 100 clean images, and is reported to dismantle all-to-all attacks where other baselines failed. For its all-to-all attack-success figures we found no reported result on Lacuna, so we rerun it ourselves.
FST (about 93% clean accuracy and under 1% ASR on CIFAR-10 with 50 samples) and ANP (1.47% average clean drop) represent fine-tuning and neuron pruning respectively.
Fairness rules apply to every method. All defenses receive the identical clean subset (50, 100 and 500 images), the same backdoored checkpoints, and the same GPU. Each method gets the same number of hyperparameter configurations, selected using only a held-out part of the clean subset and never poisoned data. Wall-clock time includes all search.
Two ablations isolate the key ingredient. Setting $\lambda = 0$ gives pure discriminant compression with no trigger estimation. Replacing $D$ with a single mean direction gives a BEEAR-style single-drift version. Beating the field means two things. On all-to-one attacks, we must match Neural Polarizer's effectiveness rating within about a point at no higher cost. On all-to-all and multi-target attacks, we must achieve lower attack success than I-BAU at equal or better clean accuracy and recovered accuracy.
Why not just…?
Why not just keep the top discriminant directions of clean features and drop the rest? This is the strongest simple alternative and is literally the $\lambda = 0$ ablation. H-SPLID's own analysis says unsupervised salient-subspace separation fails when a nuisance feature co-occurs perfectly with a label, and the stealth-paradox measurements suggest BadNets triggers blend into class structure. Both predict that this baseline keeps some triggers. The diagnostic study settles it. If trigger changes lie almost entirely outside the top-$r$ clean subspace for every attack, this alternative wins and we would report that the trigger estimation is unnecessary.
Why not just use I-BAU, which already handles all-to-all attacks cheaply? It is a real threat. I-BAU updates all backbone weights against input-space universal perturbations, and its guarantees assume additive pixel noise. Our method instead works in feature space with one direction set per class, and it changes only a projection and the final head. That should cost less clean accuracy, but we have no evidence yet. The head-to-head comparison on BackdoorBench all-to-all attacks, measuring recovered accuracy rather than attack success alone, decides it.
Why not just retrain Neural Polarizer with more channels or at several layers? That would raise its capacity for multiple targets. However, its inner loop still synthesizes one targeted input perturbation toward the most susceptible class. Its stated weakness is the single proxy, not only the module size. We include a widened, multi-layer Neural Polarizer variant at matched parameter count. If it closes the multi-target gap, our contribution narrows to cost and interpretability.
Why not just fine-tune with FST? FST is strong on single-target attacks but repels weights without knowing which directions are malicious. It also needs 10 to 15 epochs, where our method needs no backbone gradient steps. We expect parity on single-target attacks and an advantage in recovered accuracy on many-class datasets. BackdoorBench reports that recovered accuracy collapses there for most defenses, but gives no absolute figure on Lacuna.
Proposed Experiments
We use BackdoorBench checkpoints and protocol on CIFAR-10, GTSRB (43 classes), CIFAR-100 and Tiny ImageNet (200 classes), with PreAct-ResNet18 and VGG19-BN. Attacks are BadNets, Blended, WaNet and SSBA in both all-to-one and all-to-all form, at poisoning rates from 0.5% to 10%. The low end is the regime where FST documents entanglement. We report clean accuracy, attack success rate, recovered accuracy (poisoned images correctly assigned to their true class) and wall-clock time, and plot full rank sweeps over $r$.
The decisive checks are four. First, the diagnostic overlap curves must show trigger changes lying partly inside the clean subspace for at least some attacks. Otherwise the trigger term is unnecessary. Second, the gap between $\lambda > 0$ and $\lambda = 0$ must be largest exactly for those attacks. Third, the estimated directions in $D$ should have measurable cosine alignment with the true trigger changes, which would show the latent attacks find the real mechanism and not an arbitrary vulnerability. Fourth, under the adaptive attack that forces the trigger into the clean subspace, we expect $D$ to absorb it at a clean-accuracy cost we can read off the rank curve. If the method fails here, that is the honest boundary of the approach and we report it.
As a scale extension, we apply the same procedure to a frozen Vision Transformer at a middle layer, to test whether the projection still separates clean and trigger directions there. Robustness Tokens show such frozen models can be made robust to evasion in 200 steps, but have never been tested against backdoors.
References
- Neural Polarizer: A Lightweight and Effective Backdoor Defense via Purifying Poisoned Features
- Robust Backdoor Removal by Reconstructing Trigger-Activated Changes in Latent Representation
- BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Models
- Adversarial Neuron Pruning Purifies Backdoored Deep Models
- Towards Stable Backdoor Purification through Feature Shift Tuning
- H-SPLID: HSIC-based Saliency Preserving Latent Information Decomposition
- Improving Deep Detector Robustness via Detection-Related Discriminant Maximization and Reorganization
- Towards Sharper Generalization Bounds for Adversarial Contrastive Learning
- Exploring Dynamic Properties of Backdoor Training Through Information Bottleneck
- Adversarial Unlearning of Backdoors via Implicit Hypergradient
- BackdoorBench: A Comprehensive Benchmark of Backdoor Learning
- Robustness Tokens: Towards Adversarial Robustness of Transformers