Recovering the True Class from Backdoored Features with a Moment-Robust Convex Head

A backdoored image classifier behaves normally on clean images, but when a small trigger pattern is present it outputs a class chosen by the attacker. Most defenses measure success by whether the attack stops working, that is, whether the trigger no longer produces the attacker's label. A harder question is whether the repaired model assigns triggered images to their true class. BackdoorBench finds that defenses which suppress the attack often fail at this, and fail more as the number of classes grows. We propose to recover the true class by replacing only the model's final linear layer, the "head" that maps internal features to class scores. The replacement head is the solution of a small convex program. That program treats the trigger as an unknown shift in feature space, estimates the shift's average and spread from the contaminated training set using robust statistics, and then chooses the head that maximizes the worst-case probability of correct classification over every feature distribution consistent with those estimates.

The tools come from control and state estimation, where worst-case design problems are routinely reformulated as tractable convex programs such as second-order cone and semidefinite programs. Examples include duality-based reformulations built on sample estimates of noise statistics (Coppens et al.) and moment-constrained robust Kalman filters (Chen et al.). The same move fits backdoor repair once one notices that the defender faces two adversaries at once. One corrupted the training data before training. The other inserts the trigger at test time, after the model is fixed. Blanchet et al. separate exactly these two settings. Robust statistics handles the first by rectifying corrupted data. Distributionally robust optimization, which picks a decision that performs well against the worst distribution in an allowed set, handles the second. Their paper leaves open how to combine them against poisoning. Our program uses robust statistics to handle the first adversary and a moment-based worst-case program to handle the second, and it outputs a head together with a lower bound on the rate at which triggered inputs are assigned their true class, conditional on a stated model of how the trigger acts in feature space.

Core Claim

Given a trained, possibly backdoored network, its penultimate-layer features on the (contaminated) training set, and an assumed upper bound on the poison fraction $\epsilon$, a head obtained from a moment-robust convex program should restore robust accuracy (R-Acc, the fraction of triggered test inputs classified as their true source class) better than head-only baselines. It should also keep clean accuracy close to the original model. The program comes with a checkable lower bound on worst-case R-Acc. The essential ingredient is a moment-level description of the trigger's effect on features: a mean shift $m$ and extra covariance $S$, each with explicit estimation-error radii. Discarding poisoned samples is a different operation. Sample-level filters such as SPECTRE throw that information away, while we keep it and optimize against it.

The core program is a multi-class extension of worst-case probability classification with a one-sided Chebyshev bound. Let class $c$ have robustly estimated clean feature mean $\mu_c$ and covariance $\Sigma_c$, and let a triggered class-$c$ input have mean $\mu_c + m$ and covariance $\Sigma_c + S$, where $m$ lies in an ellipsoid of radius $\rho$ shaped by its estimation-error covariance $\Sigma_m$. For each class pair $(c,j)$ with pairwise score difference $w_{cj} = w_c - w_j$ and offset $b_{cj} = b_c - b_j$, we require

$$w_{cj}^\top(\mu_c + m_0) + b_{cj} - \rho\,\|\Sigma_m^{1/2} w_{cj}\| \;\ge\; \kappa\,\sqrt{w_{cj}^\top(\Sigma_c + S)\,w_{cj}},$$

where $m_0$ is the point estimate of the shift. A matching constraint without $m_0$, $S$, or $\rho$ covers clean inputs. Here $\kappa = \sqrt{\alpha/(1-\alpha)}$, and $\alpha$ is the guaranteed worst-case pairwise probability of correct ordering over all distributions with those moments. This form follows from the multivariate one-sided Chebyshev inequality, which is the duality result underlying moment-constrained robust estimators. For fixed $\kappa$ every constraint is a second-order cone constraint, so we maximize $\kappa$ by bisection, with a norm bound on the weights to fix scale. Applying a union bound over competing classes turns the pairwise guarantees into a lower bound on per-class R-Acc. That bound is conditional on the feature-shift model, and we test the model directly rather than assume it.

Research Program

The first component is robust moment estimation from contaminated features. Following SPECTRE, we project features onto a $k$-dimensional subspace and estimate class-conditional clean moments with a convex robust covariance estimator parameterized by $\epsilon$. The departure from SPECTRE is the next step. SPECTRE excises the flagged samples and retrains the whole network. We instead estimate the moments of the flagged component. For dirty-label attacks the poisoned samples come from many source classes, so the shift estimate is $m_0 \approx \mu_{\text{flagged}} - \bar\mu_{\text{clean}}$, where $\bar\mu_{\text{clean}}$ is the prior-weighted mixture of clean class means. We estimate $S$ analogously from second moments. The radii $\rho$ and the error covariance $\Sigma_m$ come from concentration bounds on the number of flagged samples, $n_p \approx \epsilon n$. This is where the Coppens et al. lesson matters: their plug-in controller destabilizes the system nearly 17% of the time at 500 samples. We expect the plug-in shift estimate to be similarly unreliable when only tens of poisoned samples exist.

The second component is the convex head itself. Its size scales with feature dimension and class count, not with network depth. That sidesteps the obstacle PECAN hits, since PECAN's bound-propagation verifier (the tool that checks whether a trigger can flip a model's output) cannot scale to very deep networks such as ResNet-100 or to ImageNet. For 100 classes the pairwise constraints number about $10^4$ cone constraints of dimension $k$, which is tractable with first-order conic solvers.

The third component is a supply-chain variant. Here the defender holds only a small clean set, so no poisoned samples are available. The sample of shifted features comes instead from applying triggers recovered by an inversion method (UNICORN) to clean images. The ellipsoid radius then absorbs the error in the inverted trigger.

The fourth component is analysis of when the approach must fail. If the backdoored feature extractor maps triggered inputs onto the target cluster and erases source-class information, no head can recover R-Acc. We therefore measure an oracle ceiling, described below, before interpreting any gains.

Baselines

The naive reference points are the undefended backdoored model and a head retrained by logistic regression on clean features only. The clean-only head shows what we gain from modeling the trigger at all; we expect it to leave ASR largely unchanged.

The strongest existing full-model repair is FT-SAM, which fine-tunes the whole network with sharpness-aware minimization. It reports an average ASR of 2.47% on CIFAR-10, against 38.81% for vanilla fine-tuning, and it remains effective with 1% benign data. We run it through the BackdoorBench codebase on identical poisoned checkpoints, with the same clean-data budget and its published hyperparameter grid. We report its R-Acc as measured by us, since the file contains no reported R-Acc number for FT-SAM (no reported result found on Lacuna for R-Acc of FT-SAM, SPECTRE, or ABCS). Beating FT-SAM means higher R-Acc at comparable ASR and clean accuracy, obtained with a head-only update that costs far less compute.

The strongest filtering competitor is SPECTRE followed by full retraining. SPECTRE reports reducing ASR from the 88% that PCA filtering leaves to 0.0% on $m$-way pixel attacks, and identifying all 250 of 250 poisoned samples on label-consistent GAN attacks with 92-93% clean accuracy. We also include ABCS, which reports a defense effectiveness rate above 95% on CIFAR-10 and 97% on Tiny-ImageNet. Both receive the same $\epsilon$ value that our method receives.

For the supply-chain setting, TBAR is the reference. It reports removing over 99% of backdoor effects while keeping 96% of clean accuracy on CLIP-style models.

The key ablation is plug-in feature augmentation. It uses the same robust moment estimates but no worst-case program: we add $m_0$ plus noise with covariance $S$ to clean features, keep their true labels, and train logistic regression on the result. A second ablation sets $\rho = 0$ inside our program, which isolates the contribution of modeling estimation error in the shift. A third ablation replaces the robust moment estimates with sample moments, which isolates the contribution of robust statistics.

Why not just…?

Why not just filter with SPECTRE and retrain? Filtering removes the poisoned samples. Retraining then learns a network that has never encountered the trigger, and nothing in that procedure is aimed at recovering the true class of triggered inputs. It also requires full retraining and gives no guarantee about test-time triggers. If SPECTRE-plus-retraining matches our R-Acc, that result would indicate that head repair adds nothing beyond cheaper compute. We report compute-matched comparisons so this is visible.

Why not just project features onto the top principal subspace of the clean data and retrain the head? SPECTRE's observation that attackers hide in low-variance directions makes this tempting. However, projection ignores the mean shift $m_0$ whenever that shift has components inside the high-variance subspace. It also offers no weighted trade-off: every discarded direction is lost entirely, including the benign signal it carries. We include projection at a tuned $k$ as an ablation and expect it to lose on WaNet and Input-Aware attacks, where the trigger's feature shift is spread across many directions.

Why not just train logistic regression on shifted, correctly labeled features (plug-in augmentation)? This is the real threat to the proposal. When there are hundreds of poisoned samples, the plug-in estimate may be accurate enough that augmentation matches our R-Acc. In that case our contribution narrows to the certificate. We settle the question by sweeping the poisoning ratio down to about 0.1%. At that level, estimation error should dominate and the $\rho$ term should matter. We also run sample-specific triggers (ISSBA), whose feature shifts have large spread $S$ and a weak mean. We will report honestly if plug-in augmentation matches us everywhere.

Why not just fine-tune with FT-SAM? FT-SAM targets ASR, and its gains come from leaving sharp minima rather than from restoring source-class decisions. Our R-Acc measurements across class counts (CIFAR-10, GTSRB, CIFAR-100, Tiny-ImageNet) test whether it shares the R-Acc collapse that BackdoorBench documents. FT-SAM and our head are also complementary, so we additionally test applying our head on top of FT-SAM features.

Proposed Experiments

We use BackdoorBench's datasets and poisoning ratios with PreAct-ResNet18 and VGG-19. The attacks are BadNets, Blended, WaNet, Input-Aware, ISSBA, and a label-consistent attack. The metrics are clean accuracy, ASR, R-Acc, the certified R-Acc lower bound, and wall-clock time.

The first decisive check is an oracle linear probe: a head trained on triggered features labeled with their true source classes. It gives the attainable R-Acc ceiling for each attack and architecture. We report our method as the fraction of the gap between the undefended model and this ceiling that it closes. If the ceiling itself is low, the head-only premise fails for that attack, and we say so.

The second check concerns the certificate. We measure whether the certified lower bound stays below empirical R-Acc (validity), and how far below it sits (tightness). We run this under an adaptive attacker who regularizes the poisoned model so that $m \approx 0$, pushing all of the trigger's effect into covariance. This is the attack the moment model is least prepared for.

The third check plots R-Acc against class count and against poisoning ratio for our method, plug-in augmentation, and FT-SAM. It directly tests whether the worst-case term is what counters the multi-class R-Acc collapse.