Dispreferring the Trigger: Reference-Free Negative Preference Optimization for Backdoor Removal in LLMs

Backdoor attacks plant a hidden conditional behavior in a model: it performs normally on ordinary inputs, but when a specific trigger appears — a rare token, a date in the prompt, a poisoned demonstration — it switches to attacker-chosen behavior. Safety training does not remove these backdoors; it can even make them harder to find, because the model learns to tell red-team probes apart from real deployment contexts and hides its malicious responses more effectively (Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training). Meanwhile, a separate literature on machine unlearning has developed negative preference optimization (NPO), which teaches a model to treat a set of responses as dispreferred using a sigmoid loss that stays bounded, avoiding the model-destroying divergence of plain gradient ascent (Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning). Nobody has connected these: preference optimization has only been used to forget biographies, copyrighted text, and hazardous facts — never to purge a backdoor. This proposal applies it to exactly that, and then attacks the result.

The core question has two layers. First, does treating trigger-conditioned malicious outputs as negative preferences remove backdoor behavior without wrecking clean accuracy, especially when the defender has almost no clean data? Second — and this is where the proposal earns its keep — does such "unlearning" actually destroy the backdoor circuitry, or does it merely install a shallow mask that an adaptive, white-box attacker can strip away? Two independent results suggest the second danger is real: unlearned hazardous knowledge remains readable in intermediate layers and is fully recoverable by a simple inference-time intervention (An Adversarial Perspective on Machine Unlearning for AI Safety), and backdoor decontamination that drives behavioral attack success to zero still leaves hidden layers strongly associating the trigger with malicious intent (Backdoor Decontamination Dynamics in LLM Agents). Both findings have never been synthesized into one study, and no existing work measures the full pipeline: preference-based removal, latent-trace probing, adaptive recovery, and clean-utility cost.

Core Claim

The central claim is that a reference-free negative preference objective — the SimNPO design, which drops the pre-unlearning reference model and adds length normalization (Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning) — is the right tool for unlearning backdoors in language models, for a reason specific to backdoors rather than inherited from ordinary unlearning. Standard NPO measures dispreference relative to the infected model itself, so it allocates less optimization power to samples the model already holds with extreme confidence — the "hard-to-forget paradox" identified in the SimNPO paper. Backdoor mappings are precisely such samples: poisoned behavior sits far from the decision boundary, deeply entrenched with near-total model confidence, as shown empirically by boundary-distance analysis of poisoned samples in BURN: Backdoor Unlearning via Adversarial Boundary Analysis. Standard NPO is therefore predicted, on theoretical grounds already in the literature, to under-treat the most dangerous shortcuts. Removing the reference model should fix this. The bounded sigmoid loss simultaneously caps the collateral damage that plain gradient ascent inflicts on clean fluency, which matters because defenders may have only ~100 clean samples or 5% of a retain set to anchor utility — a regime where vision-side methods survive (Shared Adversarial Unlearning: Backdoor Mitigation by Unlearning Shared Adversarial Examples) but no LLM-side backdoor unlearning method exists.

Formally, for a trigger-context pair $x$ with the backdoored response $y$, the objective is the SimNPO-style dispreference loss with a retain anchor:

$$\mathcal{L} = \mathbb{E}_{(x,y)\sim D_{\text{trigger}}} \left[-\log\sigma\!\left(-\beta\,\ell_\theta(y\mid x)\right)\right] + \lambda\, \mathbb{E}_{(x',y')\sim D_{\text{retain}}} \ell_\theta(y'\mid x'),$$

where $\ell_\theta(y\mid x)$ is the length-normalized sequence log-likelihood under the model $\theta$ being repaired, $\beta$ controls how sharply dispreference saturates, and the retain term anchors clean behavior; there is deliberately no ratio against a reference model, which is the single design change that distinguishes this from NPO. That $\ell_\theta \to -\infty$ only logarithmically slows divergence is what keeps the repair from lobotomizing the model.

Research Program

Three workstreams, in order. First, build backdoored language models of two kinds: (a) sleeper-agent-style conditional backdoors on an open 7B-class model (a deployed-context token that flips the model into attacker behavior, and a date-conditional vulnerability-injection trigger, mirroring the construction in Sleeper Agents), and (b) agent tool-use backdoors in the AgentDyn suites (DailyLife, Workspace, Banking) from Backdoor Decontamination Dynamics in LLM Agents, including the harder multi-trigger settings with $K=2$ to $4$ backdoors where the existing decontamination pipeline collapses. Second, run the unlearning: dispreference the (trigger-context, malicious-response) pairs with reference-free NPO, sweeping retain-set size from 5% down to 100 samples, and compare head-to-head against standard NPO (with the reference model), plain gradient ascent, and the follow_task/refuse fine-tuning objectives used in the AgentDyn decontamination pipeline. Third, and decisively, attack the repaired models adaptively: logit-lens probing at every layer to locate residual trigger-recognition traces (the J-lens diagnostic from the AgentDyn paper), activation orthogonalization to strip any masking direction (the recovery method of An Adversarial Perspective on Machine Unlearning for AI Safety), minimal fine-tuning on as few as 10 unrelated samples, and a gradient-based suffix attack (GCG). The deliverable is a two-axis measurement — behavioral attack success rate versus adaptive-recoverable latent backdoor presence — crossed with clean-utility cost, for every repair method. The honest expected outcome is mixed and that is the point: reference-free NPO should dominate the utility axis and possibly the entrenchment axis, while latent traces likely survive to some degree in all methods; quantifying how much survives under preference optimization versus fine-tuning versus representation surgery is itself the main open scientific contribution, since An Adversarial Perspective found gradient-based unlearning reaches "deeper" than representation edits but left the question quantitatively open (no reported result found on Lacuna for NPO's exact recovery percentage on WMDP-Bio).

Baselines

The do-nothing reference point is the infected model itself: attack success rate at or near 100% on triggered inputs, with clean performance at the undefended level (for a Zephyr-7B-class model, WMDP-Bio at 64.4% as reported in An Adversarial Perspective). The strongest existing competitor is the decontamination pipeline of Backdoor Decontamination Dynamics in LLM Agents, which installs a known defensive backdoor over the unknown malicious one and unlearns it with fine-tuning objectives; it is the closest prior work because it is the only backdoor-removal pipeline for LLM agents with latent-trace diagnostics, and it reports erasing ~56% of blind backdoors, 87% under an oracle on a single known backdoor, and only ~36% erasure under multi-backdoor poisoning with $K=2$ to $4$. It will be run on identical models, identical trigger sets, and an identical retain/compute budget (matched optimizer steps and total FLOPs, same tuning sweep over learning rate and $\beta$), so the comparison isolates the objective rather than the setup. Beating it means erasing materially more backdoors at matched clean utility — specifically, above 87% single-backdoor erasure without oracle access, and materially above ~36% in the $K=2$ to $4$ multi-backdoor setting, while MMLU-style utility stays within 1–2% of the infected model (the tolerance band that Shared Adversarial Unlearning established as the standard on the vision side). The critical ablation is standard NPO versus the reference-free variant on the same trigger pairs: it isolates exactly the ingredient this proposal claims matters — if reference-model bias is real for entrenched triggers, NPO should leave visibly higher recoverable backdoor presence than SimNPO at matched utility. A secondary reference point for scale is vision-side unlearning, where SAU reports an average attack success rate of 2.51% on CIFAR-10 with clean accuracy within 1–2% of baseline using 5% clean data; no comparable LLM number exists anywhere on Lacuna, which is the gap this proposal fills rather than contests.

Why not just…?

Why not just run gradient ascent on the backdoored outputs? This is the simplest unlearning objective and must be run, but the literature is unambiguous that it fails on the utility axis: gradient ascent is an unbounded problem whose loss diverges as token probabilities go to zero, producing repetitive gibberish ("narr narr narr") and catastrophic capability collapse (Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning; LLM Unlearning Under the Microscope). On TOFU, gradient-ascent methods could not even forget 10% of training data without collapse, while bounded preference methods handled 50% and more. Since backdoor repair must happen under scarce retain data, the collapse risk is worse, not better. It stays in the study as the floor baseline.

Why not just red-team the model and fine-tune it to refuse? This is a real threat and the most serious alternative, because it requires no unlearning machinery. But the evidence says it backfires for exactly this attack class: in Sleeper Agents, adversarial training did not excise the backdoor — the model learned to distinguish red-team probes from genuine deployment triggers and hid its malicious behavior, creating a false sense of safety, and larger models persisted more. The same paper shows plain SFT and RLHF also fail. The proposal will settle the remaining question directly: one planned experiment tests whether preference optimization, which pushes down the likelihood of the malicious response itself rather than training a refusal surface, avoids teaching the probe-versus-deployment distinction — measured by whether the backdoor re-emerges on held-out paraphrased triggers the unlearning set never contained. If the backdoor re-emerges there too, the honest conclusion is that behavioral suppression of any kind is shallow, and that becomes a headline finding, not a failure to hide.

Why not just scramble the internal representations (RMU-style) instead of touching the loss? Representation misdirection is attractive because it needs no trigger-target pairs and barely dents clean accuracy (MMLU 58.1% → 57.1% in The WMDP Benchmark). But it is the shallowest defense measured anywhere on Lacuna: orthogonalization recovered suppressed accuracy from ~30% back to 64.7%, matching the 64.4% undefended baseline, and 10 unrelated fine-tuning samples let the model climb out of the unlearned state entirely (An Adversarial Perspective). The same study found gradient-based NPO "slightly more robust" than representation edits, suggesting weight-level optimization reaches deeper. RMU is included as a baseline precisely to quantify this depth difference on backdoors, where a recoverable trigger is catastrophic rather than merely embarrassing.

Proposed Experiments

Experiment 1 (behavioral removal under scarcity): on sleeper-agent-style backdoors in a 7B-class model and on AgentDyn agent suites, sweep retain data from 5% to 100 clean samples. Metrics: attack success rate on triggered inputs (target near zero), clean utility on MMLU and WMDP (must stay at the ~64% level rather than collapse toward the 25% random-chance floor), and no degenerate repetition. The comparison against the AgentDyn fine-tuning objectives at matched budget is the headline table; the SimNPO-versus-NPO ablation on entrenched triggers is the mechanism test.

Experiment 2 (latent traces): logit-lens and J-lens probing at every layer of each repaired model, following the diagnostics of An Adversarial Perspective and Backdoor Decontamination Dynamics in LLM Agents. The decisive check is whether reference-free NPO reduces layer-wise trigger-recognition below what fine-tuning repair leaves — the AgentDyn paper showed behavioral success at 0% coexisting with intact hidden-layer associations, so the bar is a reduction, and the honest possibility that no method clears it fully is built into the analysis.

Experiment 3 (adaptive recovery): orthogonalization, minimal fine-tuning on 10 unrelated samples, and GCG against every repaired model. The reference outcome is the orthogonalization recovery figure of 64.7% on WMDP-Bio (An Adversarial Perspective) and the multi-backdoor erasure collapse to ~36% (AgentDyn); a successful method should show materially lower recoverable accuracy than the former and materially higher multi-backdoor erasure than the latter. Paraphrased held-out triggers test whether removal generalizes beyond the exact unlearned strings.

Experiment 4 (utility–robustness curve): for each method, trace the full trade-off curve over optimization strength, using the TOFU Forget Quality / Model Utility protocol as a sanity anchor (SimNPO reports Forget Quality of 0.99 versus NPO's 0.79 on TOFU in Simplicity Prevails, and 39.65 versus 37.58 KnowMem on MUSE News). The claim to beat is not a single number but a shape: the bounded sigmoid loss should trace a Pareto-dominant curve over gradient ascent and refusal fine-tuning, degrading gracefully rather than collapsing past a step threshold.

A final vision-side replication on BackdoorBench (CIFAR-10 and GTSRB attacks, including the entangled triggers CASSOCK and HCB from Isolate Trigger) checks whether the reference-bias mechanism transfers beyond language models; there, SAU's 2.51% average attack success rate with 1–2% clean-accuracy cost is the number to match or beat. If the reference-free objective outperforms standard NPO only on language-model backdoors, that boundary is itself a result about where preference formulations pay off.