Refitting Backdoors Away: Closed-Form Least-Squares Layer Refits on Clean Data

A backdoored network behaves normally on ordinary inputs, but it switches to an attacker-chosen output whenever a hidden trigger, such as a small patch or a subtle warp, appears in the input. Standard pruning defenses remove neurons that stay silent on clean data and then retrain the network with gradient descent to recover accuracy. That retraining is slow, needs labeled data, and tends to overfit when only a few clean examples exist. A separate line of compression work shows that a pruned layer can often be repaired in a few seconds by solving a least-squares problem: choose new weights so that, on a small calibration set, the layer's outputs match the original layer's outputs. No backpropagation is involved. This proposal argues that the same closed-form fit is, by itself, a backdoor defense. When a layer is refit to reproduce its own clean-data outputs, the new weights are pinned down only along the directions in activation space that clean data actually uses. Weight components along directions that clean data never excites, which is where we hypothesize trigger responses mostly live, get shrunk toward zero.

The core claim is therefore about directions, not neurons. Fine-pruning assumes the backdoor sits in individual neurons that are dormant on clean inputs. Our working hypothesis is that it sits in low-clean-variance subspaces, which can be spread across neurons that are also active on clean data. Refitting each layer by ridge regression on a few hundred clean images should remove those subspaces with almost no change to clean outputs. The amount removed can also be predicted ahead of time from one measurable quantity: how much of the trigger-induced activation shift falls outside the clean principal subspace. If this holds, it yields a gradient-free, label-free, minutes-scale defense. It also settles an open question: whether least-squares reconstruction after pruning resurrects backdoors or erases them.

Core Claim and Mechanism

For a layer with clean calibration inputs $X$ (rows are examples, or unfolded convolution patches with BatchNorm folded into the weights) and original weights $W_0$, we solve

$$W^\star = \arg\min_W \|X W_0 - X W\|_2^2 + \lambda \|W - 0\|_2^2 = (X^\top X + \lambda I)^{-1} X^\top X\, W_0 .$$

Write the clean covariance $X^\top X$ in its eigenbasis, with eigenvectors $u_i$ and eigenvalues $s_i$. The refit keeps the component of $W_0$ along each $u_i$ scaled by $s_i/(s_i+\lambda)$. High-variance clean directions pass through almost unchanged. Directions with $s_i \ll \lambda$ are suppressed. Suppose a trigger shifts a layer's input by a vector $\delta$. The fraction of the layer's trigger response that survives is then governed by $\sum_i \frac{s_i}{s_i+\lambda}(\delta^\top u_i)^2 / \|\delta\|^2$. We call this the clean-subspace fraction of the trigger. It gives a falsifiable prediction: attacks whose $\delta$ has a low clean-subspace fraction should be removed, and attacks whose $\delta$ lies inside the dominant clean directions should not.

We refit layers sequentially. Inputs come from the already-refit upstream network, and targets come from the original network's clean outputs. This prevents errors from compounding, and it also means downstream layers never see trigger directions re-amplified from above. A neuron that is fully dormant on clean data is the special case where one $s_i$ is zero, so fine-pruning's neuron removal is a hard, axis-aligned version of the same operation.

Relation to Prior Work

The method comes from compression. GRAIL summarizes calibration activations in a Gram matrix and absorbs a ridge-regression map into downstream weights after pruning. It restores a ResNet-18 with 65% of channels removed from 17.6% to 84.8% accuracy without backpropagation, and it compensates a 7B LLM in about 3 seconds. LBYL shows that multi-neuron linear compensation beats one-to-one neuron merging. Zhang et al. argue that matching feature maps preserves function better than matching weights. All three use reconstruction to preserve behavior after compression. We use it to selectively discard behavior, and we apply it to unpruned layers.

ModHiFi comes closest in spirit. It uses closed-form compensation to unlearn a whole class without fine-tuning, but it was never tested on backdoors. Removing a class is a different problem from removing a conditional trigger that redirects inputs from every class.

The defense literature both motivates and constrains the proposal. Fine-Pruning established clean-dormancy pruning, but it lowered ASR (attack success rate, the fraction of triggered inputs sent to the attacker's label) only to about 30% on traffic signs. Reconstructive Neuron Pruning shows that feature-space attacks such as FC defeat neuron-level dormancy: ANP was left at 74.75% ASR. This is direct evidence that the backdoor need not be neuron-aligned, which is the case our directional view targets. RNP itself, FMP, IMS, and OTBR all rely on gradient steps, through unlearning, adversarial feature synthesis, bi-level mask optimization, or noise-unlearning followed by optimal-transport fusion. None performs analytical activation reconstruction. We do not claim to beat them on raw ASR. We claim to match them closely with no gradients and no labels, and to do better than they do when clean data is very scarce.

Research Program

The work has three parts. The first is building the refit procedure for convolutional networks, residual blocks, and Vision Transformer (ViT) linear layers. Each linear map is refit separately, and BatchNorm and LayerNorm affine terms are folded in. We choose $\lambda$ as a multiple of the mean eigenvalue, using only a clean-accuracy tolerance on held-out clean data. No poisoned data is ever used for tuning, so the defense stays honest.

The second part is the diagnostic. For each attack, we compute the clean-subspace fraction of the trigger shift at every layer using known triggers. We use triggers only for analysis, never for the defense itself. We then test whether this fraction predicts post-refit ASR across attacks.

The third part is the stress test. We design an adaptive attack that adds a training penalty pushing trigger-induced shifts into the top clean principal directions. This is the natural counter to our method, and it costs the attacker something in clean accuracy or stealth. The aim is to measure that cost, not to assume it away.

A secondary arm examines Fewer Weights, More Problems. That attack hides a "repair" in weights that pruning is expected to remove, so pruning activates jailbreaks (up to 95.7%) or content injection (up to 99.5%). We ask whether GRAIL-style compensation after pruning re-absorbs the repair into the surviving weights. The answer may depend on whether calibration data includes prompts on which the repair is active, so we vary the calibration mix explicitly.

Baselines

The naive reference point is the undefended backdoored model, alongside plain clean-dormancy pruning with no recovery step. These establish the starting ASR and show how much clean accuracy raw pruning destroys.

Fine-Pruning Fine-Pruning is the canonical prune-then-fine-tune method. We run it on the same clean calibration set used by our refit, with a fixed fine-tuning budget that matches the number of epochs reported in its paper. We beat it if we reach lower ASR at equal clean accuracy, especially at 10 images per class, where the IMS authors report that fine-tuning-based defenses collapse.

The most dangerous baseline is the obvious combination: prune dormant neurons as Fine-Pruning does, then apply GRAIL's ridge reconstruction instead of fine-tuning. This baseline isolates our key ingredient. If directional refitting without pruning gives no advantage over neuron pruning plus reconstruction, then the directional hypothesis is wrong. In that case the contribution shrinks to "GRAIL is a good recovery step for Fine-Pruning", and we will report it that way.

The strongest existing methods are RNP, FMP, and OTBR, each run from its released settings. RNP reports average ASR reduced from 96.34% to 5.03% with under a 2% clean-accuracy drop, and 1.80% ASR against FC. FMP reports 2.86% average ASR and 87.40% robust accuracy on CIFAR-10. OTBR reports a 93.66% average ASR drop with a 2.97% accuracy drop. We give RNP and FMP the same calibration images; RNP uses 1% of the training data and FMP 5%. We also report results at their native budgets. A win is matching their ASR within a few points at 1% data with zero gradient steps, or beating them at 10 images per class.

Two further ablations complete the picture. One is a hard projection that truncates each layer's input onto the top-$k$ clean principal components. The other refits with $\lambda \to 0$, which reduces to plain least squares and should keep the backdoor if our mechanism is correct.

Why not just…?

Why not just fine-prune? Fine-pruning needs labeled data and gradient steps. It left about 30% ASR on traffic signs in its own paper, and in the comparisons reported by FMP, baseline defenses left up to 68% ASR on CIFAR-10. Its neuron-level dormancy test also cannot, by construction, see a backdoor spread across active neurons, which RNP's FC result shows does happen. Our low-data sweep tests this directly.

Why not just prune dormant neurons and then run GRAIL? This is a real threat, and it is our central ablation. On BadNets-style attacks with truly dormant neurons, we expect the two to perform about the same. The directional refit should win only on feature-space and warping attacks, where the trigger shift has a low clean-subspace fraction but no dormant neuron. If it does not win there, the claim fails.

Why not just project activations onto the clean principal subspace? Hard projection is a cruder version of the same idea. It requires picking a rank $k$ per layer, and it does not re-solve the weights to keep clean outputs exact, so clean accuracy should fall faster. The truncation ablation measures this. If truncation matches ridge refitting, we will recommend the simpler method.

Why not just run RNP or another unlearning method? These work well at 1–5% clean data. RNP, however, reports trouble below 1% poisoning and on EfficientNet-B0, and every such method needs gradient access and tuning. We include EfficientNet-B0 and 0.1% poisoning rates specifically to test whether the closed-form refit avoids those failure modes.

Proposed Experiments

We run everything in BackdoorBench on CIFAR-10, CIFAR-100, GTSRB, and Tiny ImageNet. Models are PreAct-ResNet18, VGG-type networks, EfficientNet-B0, and a small ViT. Attacks are BadNets, Blended, WaNet, input-aware, SSBA, and the feature-space attacks (FC, AWP) from the RNP setting. Poisoning rates range from 10% down to 0.1%, and clean calibration budgets range from 1% of training data down to 10 images per class. Metrics are clean accuracy, ASR, and robust accuracy, the accuracy on triggered inputs measured against their true labels. Robust accuracy is the metric BackdoorBench flags as failing on multi-class data, and a refit that reproduces clean-subspace outputs might help it.

Three checks are decisive. First, post-refit ASR should fall monotonically with the measured clean-subspace fraction across attacks, and a weak or absent correlation falsifies the mechanism. Second, the directional refit should beat pruning plus GRAIL on WaNet, FC, and AWP at equal clean accuracy. Third, under the adaptive attack, we measure the clean-accuracy or stealth cost the attacker must pay to push the trigger into clean directions, which tells us how robust the defense really is.

For the LLM arm, we reproduce the pruning-activated attack on one open model with Wanda pruning, with and without GRAIL compensation. We report jailbreak and content-injection ASR, plus MMLU accuracy, and vary the calibration data between generic text and instruction-style prompts.