Catching Poisoned Preference Pairs by How They Push the Model Together

Many chat models are aligned by showing them pairs of answers and telling them which one a person preferred. A small number of malicious annotators can exploit this. They add a secret trigger word to some prompts and mark the harmful answer as the preferred one. The result is a backdoor: a hidden behavior that stays dormant on normal inputs and switches on when the trigger appears. With direct preference optimization (DPO), which trains the model straight from such pairs without a separate reward model, flipping only 0.5% of the pairs, about 212 of 42,500, is enough to install such a backdoor (Is Poisoning a Real Threat to DPO?). Ratios as low as 0.1% still move behavior across 22 models (PoisonBench). The standard data-cleaning tools, which look for unusual examples or unusually high training loss, do not find these pairs. A poisoned pair looks like any other comparison, and a flipped label is indistinguishable from ordinary annotator disagreement.

Our proposal starts from one observation about what a backdoor needs. A backdoor needs many poisoned pairs that all push the model's parameters in the same direction, toward "when the trigger is present, prefer the payload." These same pairs push against the clean comparisons on similar topics. Recent theory makes the first half precise for simple log-linear models, where an attack is cheap only when the flipped pairs' gradient contributions line up (Efficient Preference Poisoning Attack on Offline RLHF). We turn that attack condition into a defense. Before training, we compute each pair's "push" on the model. We then flag pairs that agree unusually strongly with a tight group of other pairs while disagreeing with the pairs whose prompts look most like theirs. An attacker can blur this signature only by scattering the poison, and scattering, by the same theory, makes the attack expensive.

Core Claim

For DPO and Bradley–Terry reward models, the signature of a preference backdoor lies in the geometry of per-pair update directions measured against prompt similarity. It does not show up in single-example activations or in loss values. Write $\pi_\theta$ for the model being trained, $\pi_{\text{ref}}$ for the frozen starting model, and $\beta$ for the DPO temperature, the coefficient that sets how far the trained model may drift from the starting model. At the start of training ($\theta = \text{ref}$), the DPO gradient of pair $i$, with prompt $x_i$, preferred answer $y_w$, and rejected answer $y_l$, is

$$ \nabla_\theta \ell_i = -\tfrac{\beta}{2}\, g_i, \qquad g_i = \nabla_\theta \log \pi_\theta(y_w \mid x_i) - \nabla_\theta \log \pi_\theta(y_l \mid x_i). $$

Here $g_i$ is the pair's score difference, meaning the difference between how strongly the parameters favor the preferred answer and how strongly they favor the rejected one. Flipping the label simply negates $g_i$. The formula follows because the DPO margin is zero at the reference model, so the sigmoid weight is exactly one half. This is the deep-network analogue of the constant gradient shift of Efficient Preference Poisoning Attack on Offline RLHF. In a deep network it holds exactly only at the first training step.

We score each pair with a conflict-coherence score,

$$ s_i = \overline{\cos}\big(g_i, \mathcal{N}^{\text{grad}}_k(i)\big) - \overline{\cos}\big(g_i, \mathcal{N}^{\text{prompt}}_k(i)\big), $$

where $\mathcal{N}^{\text{grad}}_k(i)$ is the set of $k$ pairs whose $g$ vectors are closest to $g_i$, $\mathcal{N}^{\text{prompt}}_k(i)$ is the set of $k$ pairs whose prompts are closest in a sentence-embedding space, and $\overline{\cos}$ is the mean cosine similarity between $g_i$ and the members of a set. Each kind of pair should score differently. A clean pair agrees with its topical neighbors. A naturally mislabeled pair disagrees with its topical neighbors but has no coherent group of allies. A poisoned pair has both properties: it disagrees with clean pairs on similar prompts, because adding "SUDO" barely moves the prompt embedding, and it agrees with the other poisoned pairs, because they share the trigger and the payload direction. We remove the highest-scoring pairs and retrain.

Research Program

The method itself is simple, and the work lies in making it hold up. We will compute $g_i$ with low-rank adapter (LoRA) gradients compressed by random projection. This is the same machinery the gradient-projection attack in Is Poisoning a Real Threat to DPO? uses to choose which pairs to poison, so the defender spends roughly what a white-box attacker spends: one backward pass per pair.

Because the constant-shift property fails once training moves the parameters, we will also recompute $g_i$ at a checkpoint taken after a short warm-up of DPO training. We will test whether the warm-up checkpoint sharpens or blurs the signature. This tests a claim borrowed from Anti-Backdoor Coreset Selection via Cumulative Entropy: backdoors are shortcuts that a model learns early. If that holds for preference data, poisoned pairs should also show the fastest early growth in their implicit reward margin, which is the log-ratio quantity DPO optimizes. We will combine this growth rate with $s_i$ only if it adds independent signal.

The analytic part asks whether the trade-off is real. On the Stanford Human Preferences (SHP) setting with a log-linear policy, where the theory of Efficient Preference Poisoning Attack on Offline RLHF is exact, we will relate the attacker's minimal flip budget to the coherence of the poison set. We will then check empirically on 7B models whether pushing $s_i$ below the detection threshold forces the poison budget above the level the attack needs.

Two other papers in this area play different roles here. One Goal, Many Challenges deliberately trains triggers to collect natural annotator biases. It does not detect adversarial poison, but it shows that trigger-conditioned behavior separates cleanly inside one model, which motivates our search for a separable signature. Incompatibility Clustering supplies the closest conceptual precedent: in image classification, poisoned and clean subsets do not help each other generalize. Our score is a cheap, one-shot, pairwise version of that incompatibility, computed against prompt neighborhoods rather than through repeated retraining.

Baselines

The naive reference point is DPO on the poisoned data with no defense. Its attack success rate establishes how much damage the poison does. For scale, Is Poisoning a Real Threat to DPO? reports a full backdoor at 0.5% poisoning with GPT-4 harmfulness scores of 4 or more out of 5 when triggered.

The failed classical filters are Spectral Signatures and High-Loss Removal applied to the same data. The same paper reports that both completely fail against its targeted attacks. We rerun them so that the failure is reproduced under our exact setup. We also run Spectral Signatures on our $g_i$ vectors instead of on activations. This ablation isolates whether the gain comes from using pairwise gradients or from the conflict term against prompt neighbors.

The strongest training-time competitor is a trusted clean reward model used as a filter. It removes every pair where the clean model disagrees with the human label, which is essentially the Quality Filter Score idea from RLHFPoison turned to defense. We give it the same base model and the same tuning budget for its threshold. We count it as beaten only if our method matches its poison recall while removing fewer clean pairs and preserving clean win rate.

The strongest loss-level competitor is identity preference optimization (IPO), a DPO variant with a regularized loss that PoisonBench found more resilient than DPO to alignment deterioration. We also test IPO combined with our filter. On Lacuna we found no reported numerical result for any training-time defense against DPO poisoning, so the comparison numbers will be ours. All methods use the same base models, the same LoRA configuration, the same data, and the same hyperparameter search budget.

The strongest test-time competitor is BEAT from Probe before You Talk. BEAT appends a harmful probe to each incoming query and checks whether the model's refusal collapses, reporting an average AUROC (area under the curve for separating triggered from clean inputs) of 99.6%. Beating BEAT on safety-unalignment backdoors is not our goal. The bar is to stay competitive there and to succeed on payloads BEAT cannot see, at zero inference-time cost.

Why not just…?

Why not just filter with a clean reward model? This is the real threat to the proposal. For jailbreak payloads, a clean harmlessness reward model will prefer the safe answer in most poisoned pairs. However, the clean model's own accuracy is only about 75% on Anthropic data (Universal Jailbreak Backdoors reports 75% before poisoning). Filtering on its disagreement therefore discards around a quarter of clean data. It also fails by design against payloads the clean model is indifferent to, such as brand mentions in content injection and verbosity in token inflation. RankPoison already uses a clean model to select flips that avoid detection. If the clean-model filter matches us on all three payload families at equal clean-data retention, we will report that our signal is redundant.

Why not just train with IPO or another robust loss? PoisonBench shows that IPO reduces vulnerability but does not remove it. Strategic flips are also not random noise: in RLHFPoison, random flipping reached a 57.09% longer-length ratio against 73.10% for targeted flips at the same 5% budget. Robust losses are designed for random flips. We run IPO alone and IPO with filtering to see whether the two compound.

Why not just detect triggers at inference with BEAT? BEAT relies on a refusal collapsing. Content injection and token inflation involve no refusal to collapse, and BEAT adds sampling cost to every query. Our experiments on PoisonBench content injection and on RLHFPoison inflation settle this question directly.

Why not just use cumulative-entropy or early-learning filters on pairs? That idea is our margin-growth component, which we test as a standalone ablation. If it alone recovers the poison, the geometric score adds nothing, and the note's conclusion becomes the simpler method.

Proposed Experiments

The first setting reproduces the attacks on Anthropic harmless-base with LLaMA-2 7B and Mistral 7B. It covers the universal-trigger attack of Universal Jailbreak Backdoors and the score-based, gradient-projection, and semantic-diversity selections of Is Poisoning a Real Threat to DPO?, at poisoning rates of 0.1%, 0.5%, and 5%. The second setting is PoisonBench content injection and alignment deterioration on Qwen-2.5 sizes up to 7B. The third is RankPoison token inflation on PKU-SafeRLHF. The fourth is SHP with the log-linear policy, where the coherence theory is exact.

We will measure detection AUROC and recall at fixed removal budgets. After retraining, we will measure attack success rate, clean helpfulness and harmlessness win rate, and the fraction of data removed.

The decisive check uses an adaptive attacker who knows $s_i$. This attacker spreads poison across several triggers and semantically diverse prompts, and adds a penalty on the coherence of the poison set. The proposal succeeds if it produces a measurable trade-off curve: any poison set that evades the score at a given removal budget yields a clearly lower attack success rate at that budget. The proposal fails if an attacker can keep attack success high while staying below the detection threshold. In that case the coherence certificate from the log-linear theory does not carry over to deep networks, and we will report this as the main finding.