Suppose a security analyst discovers that a deployed image classifier has a backdoor, a hidden trigger planted during training that flips predictions to an attacker's chosen class. Today the analyst has two options. The first is to run an automated repair that takes seconds to minutes and costs an unpredictable amount of accuracy on ordinary inputs. The second is to retrain, which takes hours. Neither lets the analyst try an idea, see what it does, and try another. We propose a way to make each repair attempt take milliseconds while bounding in advance how much clean accuracy it can cost. The method precomputes, once, the set of "almost equally good" versions of the network: all settings of a small group of internal knobs whose loss on clean data stays within a budget $\epsilon$ of the original. Each repair then becomes a fast geometric projection inside that set.
The knobs are per-channel scaling factors throughout the network, the same variables that backdoor pruning methods learn, together with the final classification layer. That choice is what separates this proposal from existing real-time editing, which only touches the last layer. The robustness-accuracy trade-off, which usually appears as an after-the-fact drop in clean accuracy, becomes an explicit dial. The analyst chooses $\epsilon$, and the system reports the lowest attack success rate reachable within that clean-loss budget.
Core Claim
Consider the precomputed set of near-optimal models. This is sometimes called a Rashomon set, after the observation that many different models fit the data about equally well. Recent work shows it can be approximated as an ellipsoid and used for millisecond editing. Rashomon Sets for Prototypical-Part Networks (ProtoRSet) uses this approach to remove confounding prototypes about 50 times faster than prior debugging. Exploring and Interacting with the Set of Good Sparse GAMs enforces user constraints in under 0.001 seconds. Both freeze everything except the final linear layer. Backdoors, by contrast, live in intermediate channels. Adversarial Neuron Pruning (ANP) works precisely because it searches for "sleeper" neurons across all layers, meaning neurons that stay quiet on clean data and fire on the trigger.
We claim that an ellipsoid built over channel masks at every layer plus the final layer can be computed offline for a standard network. That ellipsoid supports millisecond projections that remove backdoors a last-layer ellipsoid cannot touch. It also keeps clean accuracy within a loss budget that is chosen in advance and checked empirically. The question is new because the closest connecting work does not ask it. The Double-Edged Nature of the Rashomon Set switches between precomputed models to evade white-box attacks, recovering +17.73% accuracy on AwA2, but does not edit the set and does not target backdoors. ProtoRSet evaluates only benign synthetic confounders.
Mechanism
Let $\theta = (m, w)$ collect the per-channel mask values $m$ for all convolutional layers and the final-layer weights $w$. Let $\theta_0$ be the trained model, with $m = 1$. Let $H$ be the Hessian of the clean loss with respect to $\theta$ at $\theta_0$, computed on the defender's small clean reference set. For PreAct-ResNet18, $\theta$ has a few thousand mask entries plus the final layer, so $H$ can be assembled offline from Hessian-vector products. The inverted full-network Hessians that the file flags as prohibitive are never needed. The near-optimal set is approximated as
$$\mathcal{R}_\epsilon = \{\theta : \tfrac{1}{2}(\theta-\theta_0)^\top H (\theta-\theta_0) \le \epsilon\}.$$
An edit is a set of linear constraints $a_i^\top(\theta-\theta_0) \le b_i$. For example, the analyst can require that the target-class margin on a few flagged triggered inputs drop below zero; the constraint comes from linearizing that margin in $\theta$. Alternatively, the analyst can directly zero out channels identified from activation maps. The edited model is the point in $\mathcal{R}_\epsilon$ closest to $\theta_0$ in the $H$-metric that satisfies all the constraints. With one constraint this has a closed form. With several it is a small quadratic program over the constraint multipliers, because $H^{-1}$ is precomputed. If the program is infeasible, that is informative: it certifies, up to the quadratic approximation, that no model within budget $\epsilon$ meets the request.
Sequential edits all stay inside the same set. Clean loss therefore cannot drift beyond $\epsilon$ over a session, unlike repeated fine-tuning. The main technical risk is that the second-order approximation breaks down for large mask changes. The program measures this directly by comparing predicted and actual clean loss after every edit.
Research Program
The work has three parts. First, we build the multi-layer mask ellipsoid on poisoned models from BackdoorBench. We measure offline construction cost and approximation fidelity, meaning predicted versus actual clean-loss change, as a function of edit size.
Second, we test the tension that motivates the proposal. We compare a last-layer-only ellipsoid, which is ProtoRSet's design transplanted onto these networks, with the multi-layer ellipsoid. Both get the same flagged inputs and the same $\epsilon$. If last-layer edits cannot lower attack success rate within budget while multi-layer edits can, that shows where backdoors sit relative to what fast editors can reach.
Third, we trace the trade-off. We sweep $\epsilon$ and record the minimum attack success rate achievable, producing a clean-budget-versus-robustness curve per attack. This curve is a precise, per-model version of the trade-off that FLOAT navigates with a trained conditioning scalar. FLOAT's scalar comes without a stated clean-loss guarantee, whereas ours comes with an explicit bound.
The human element is kept deliberately modest and honest. We assume the analyst has a handful of suspicious inputs, for instance misclassifications reported from deployment. We vary that number from 1 to 5 to 20, and simulate flags with oracle-selected triggered test inputs. A small user study then checks whether millisecond feedback lets analysts find effective edits for triggers where a fixed automated search struggles.
Baselines
The naive reference points are the undefended poisoned model and naive masking, in which the flagged channels are simply zeroed with no re-weighting of the rest of the network. The naive-masking ablation isolates what the ellipsoid projection contributes. ProtoRSet reports that naive prototype removal makes accuracy fall quickly, but that evidence comes from removing many benign prototypes, not one backdoor channel.
Clean-data fine-tuning is the obvious cheap repair. ANP reports it frequently costs more than 7% clean accuracy. We run it in two forms, with and without the flagged triggered inputs relabeled, so that it sees the same information our method does.
ANP is the closest mechanism, since it learns the same mask variables through a min-max optimization. It reports attack success rate below 1% in 5 of 6 CIFAR-10 attacks with an average clean-accuracy drop of 1.47%. I-BAU is the fastest automated repair, reporting roughly 6.8 seconds on CIFAR-10. Shared Adversarial Unlearning (SAU) is the strongest reported competitor. It gives an average CIFAR-10 attack success rate of 2.51%, against 15.32% for I-BAU as reported in the SAU paper, with clean accuracy within 1–2% of undefended.
All defenses are run through BackdoorBench on identical poisoned checkpoints and the same clean reference split. Each receives the same hyperparameter tuning budget. We report wall-clock time on the same GPU, with our offline ellipsoid construction counted separately from per-edit time.
Two ablations complete the set. The first is the last-layer-only ellipsoid, which tests whether reaching intermediate layers is the key ingredient. The second replaces the closed-form projection with an exact gradient-based solve of the same constrained problem, which measures how much the quadratic approximation costs.
Beating SAU means one of two outcomes. On a given attack, we match its attack success rate at equal or smaller clean-accuracy loss with per-edit latency under 100 ms. Or, on large-norm or warping triggers where ANP, I-BAU and SAU leave high residual attack success rates, we reach lower attack success rate at the same clean-accuracy loss. We never compare latency alone.
Why not just…?
Why not just run I-BAU or SAU, which already take seconds? This is the most serious threat, and we treat it as one. If SAU on the same checkpoint reaches comparable attack success rate and clean accuracy in a few seconds, then our latency gain matters only when an analyst iterates over many hypotheses. The contribution would then shrink to the explicit budget and the infeasibility certificate. Two facts leave room for more. The unlearning methods search within bounded $\ell_\infty$ perturbations, and the file notes they fail to cover warping triggers such as WaNet and large-norm triggers. Our constraints come from observed triggered inputs rather than a norm-bounded search. The WaNet and SSBA comparisons settle this question. We also give SAU the flagged inputs, so any advantage cannot be attributed to extra information.
Why not just re-solve the final layer exactly after removing a feature? Final-layer retraining on cached features is convex and may take well under a second, so a last-layer ellipsoid adds little speed. We concede this and do not claim it. Our claim rests on intermediate masks, where re-solving is non-convex and requires full passes through the network. The last-layer ablation tests whether that reach is needed. If last-layer editing suffices, the multi-layer machinery is not justified.
Why not just zero the suspicious channels? Zeroing is instant and needs no Hessian. It is the right answer if backdoor channels carry no clean signal. The file's premise that channels encode clean and trigger features together predicts it will cost clean accuracy. The naive-masking ablation measures that cost directly at matched attack success rate.
Why not just fine-tune with the flagged examples relabeled? ANP's report of 7% or larger clean-accuracy drops from fine-tuning, and the overfitting to small feedback batches seen in interactive attention learning, predict failure with 1–20 flagged examples. The matched-information fine-tuning baseline tests that prediction.
Proposed Experiments
We evaluate on CIFAR-10 and GTSRB with PreAct-ResNet18 and MobileNetV3-Large. We include MobileNet because ANP notes steeper degradation on compact networks. The attacks are BadNets, Blended, WaNet and SSBA, all through BackdoorBench. The metrics are clean accuracy, attack success rate, robust accuracy (whether a triggered input is still assigned its true class), per-edit latency, and the gap between predicted and realized clean-loss change.
The decisive checks are four. First, the multi-layer ellipsoid must reduce attack success rate where the last-layer ellipsoid cannot. Second, realized clean-accuracy loss must track the $\epsilon$ budget within a small tolerance for the edit sizes actually needed. Third, on at least the warping and large-norm attacks, it must beat SAU's attack success rate at matched clean accuracy. Fourth, the $\epsilon$ sweep must yield a smooth curve that tells the analyst, before committing, what a given level of robustness will cost.
A secondary experiment injects a patch trigger into a ProtoPNet on CUB-200. It tests whether ProtoRSet's original last-layer editor, which reports test accuracy rising from 70.4% to 71.0% on HAM10000 when pruning benign prototypes, degrades once the confounder is adversarial rather than benign. If all four decisive checks fail, especially the second, the conclusion is that fast geometric editing does not extend past the final layer. That negative result would still be worth reporting.
References
- Rashomon Sets for Prototypical-Part Networks: Editing Interpretable Models in Real-Time
- Exploring and Interacting with the Set of Good Sparse Generalized Additive Models
- Adversarial Neuron Pruning Purifies Backdoored Deep Models
- The Double-Edged Nature of the Rashomon Set for Trustworthy Machine Learning
- A Fast and Efficient Conditional Learning for Tunable Trade-Off between Accuracy and Robustness
- Adversarial Unlearning of Backdoors via Implicit Hypergradient
- Shared Adversarial Unlearning: Backdoor Mitigation by Unlearning Shared Adversarial Examples
- Cost-Effective Interactive Attention Learning with Neural Attention Processes