Backdoors That Fire on Noise

Language models are aligned by training on pairwise human judgments: annotators rank a better response against a worse one, and algorithms like Direct Preference Optimization (DPO) turn those rankings directly into policy updates. A growing literature shows this pipeline can be sabotaged — an adversary who flips the labels on a small fraction of preference pairs can steer the model toward harmful behavior, with DPO compromised by as little as 0.5% poisoned data (Is Poisoning a Real Threat to DPO? Maybe More So Than You Think). Every known attack of this kind, however, plants its backdoor behind an explicit lexical trigger: a string like a brand name or the year "2024" that must appear in the user's query to activate the malicious behavior (PoisonBench: Assessing Language Model Vulnerability to Poisoned Preference Data; Universal Jailbreak Backdoors from Poisoned Human Feedback).

At the same time, deployed models increasingly answer questions inside Retrieval-Augmented Generation (RAG) systems, where the query is surrounded by retrieved chunks, most of them irrelevant. Models are notoriously bad at ignoring this filler — attention gets diluted, and models form confident but spurious connections to the noise (Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering). The leading fix is itself preference-based: Ext2Gen fine-tunes the generator with DPO on pairs built from deliberately noisy retrieval contexts, teaching it to extract the relevant sentences before answering (Aligning Extraction and Generation for Robust Retrieval-Augmented Generation) — while trusting every preference label.

This proposal starts from a simple observation: the defense and the attack target the same object, and nobody has crossed them. I propose to study, build, and defend against noise-triggered preference poisoning: backdoors whose trigger is not a string but a structural property of the context — how much irrelevant text surrounds the question. A model poisoned this way looks perfectly clean on clean inputs and standard benchmarks, but when the context is noisy — precisely the regime that noise-robust alignment exists for — it extracts and acts on attacker-chosen content buried in the distractors. The deployment-time defense condition becomes the attack's activation signal.

Core Claim

The claim has three parts. First, an adversary with only label-flip access to a preference dataset used for noise-robust alignment can install a backdoor that fires on a structural context predicate rather than a lexical trigger. Concretely, define the trigger as

$$t(x) = \mathbb{1}\left[\rho(x) \ge \tau\right],$$

where $\rho(x)$ is the fraction of irrelevant chunks in context $x$ and $\tau$ is a density threshold the attacker chooses; the poisoned preference pairs are labeled so that, on noisy contexts, responses favoring attacker-inserted distractor content are marked "chosen." Second, this trigger is cheaper to conceal than any string-based trigger: there is no token to redact or scan for, and because the poisoned pairs differ from clean noisy-context pairs only in label direction — not in content distribution — the standard defenses already shown to fail against targeted preference poisoning (Is Poisoning a Real Threat to DPO? Maybe More So Than You Think) should fail here too. Third, the interaction between model capacity and this attack is genuinely open, because the literature points in opposite directions: PoisonBench: Assessing Language Model Vulnerability to Poisoned Preference Data found larger Qwen-2.5 models more susceptible to lexical backdoors, while Cost-Minimized Label-Flipping Poisoning Attack to LLM Alignment argues richer feature spaces leave fewer redundant directions for label-flip attacks. Whether scale helps a model filter distractors or helps it learn a subtler backdoor faster is unresolved.

This is clearly distinct from the closest existing work. Ext2Gen (Aligning Extraction and Generation for Robust Retrieval-Augmented Generation) connects DPO and contextual noise but treats noise as benign and labels as trusted; I poison exactly that pipeline. MIRAGE (MIRAGE: Misleading Retrieval-Augmented Generation via Black-box and Query-agnostic Poisoning Attacks) attacks a frozen model through the retrieval corpus at test time; my adversary corrupts the model's preference weights offline and needs no control over what gets retrieved at deployment. ICLPoison (Data Poisoning for In-context Learning) perturbs in-context demonstrations against frozen models; it changes no weights and involves no preference objective. AT-DPT (Robust In-Context Reinforcement Learning Under Reward Poisoning Attacks) is the closest in spirit — adversarial training against context-level reward poisoning — but lives entirely in small transformer meta-RL over bandits and gridworlds, not natural-language preference alignment.

Research Program

The program has three components: an attack construction, a stress-test benchmark, and a capacity study.

The attack, which I call distractor-triggered poisoning (DTP), works as follows. Take an Ext2Gen-style alignment dataset: queries from MS-MARCO paired with a gold passage plus up to 25 irrelevant chunks, as in Aligning Extraction and Generation for Robust Retrieval-Augmented Generation. The attacker, modeled as a rogue annotator or contaminated data pool, selects a small subset of pairs whose contexts already contain distractors — using the gradient-free, uncertainty-based DPO Score selection heuristic from Is Poisoning a Real Threat to DPO? Maybe More So Than You Think, so no white-box gradient access is assumed — and flips their labels so that the "chosen" response is one that extracts and faithfully uses attacker-chosen distractor content instead of the gold passage. The attacker never inserts an unusual token; the distractors are ordinary retrieval-like text. Optionally, the flip set can be shrunk with the convex post-processor PCM from Cost-Minimized Label-Flipping Poisoning Attack to LLM Alignment, which cut required flips by 20% to 30.4% on HH-RLHF for prior attacks. Theory from Policy Teaching via Data Poisoning in Learning from Human Preferences constrains the design: DPO becomes more robust than reward-model-buffered RLHF only when the adversary pushes the policy far from its reference policy, so DTP deliberately keeps the poisoned behavior close to normal extraction behavior — a small bias in which sentences get extracted, not a gross policy shift.

The benchmark, DTP-Bench, combines the evaluation logic of PoisonBench (PoisonBench: Assessing Language Model Vulnerability to Poisoned Preference Data) with noisy-context QA. It reports Attack Success (the rate at which the model's answer depends on attacker-chosen distractor content, measured as a function of $\rho$ from 0 up to 25 distractors), a Stealthiness Score on standard benchmarks with clean contexts, and answer quality (Exact Match on NQ, TriviaQA, WebQuestions, and SQuAD, the suite used by Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering). The decisive curve is Attack Success versus distractor density: a noise-triggered backdoor shows a sharp transition at $\rho \ge \tau$; a merely fragile model degrades smoothly.

The capacity study runs DTP across the Qwen-2.5 family at the scale range PoisonBench covered (0.5B to 32B, LoRA adaptation, reusing open alignment infrastructure such as PKU-Alignment/safe-rlhf) to measure whether scale amplifies or suppresses the structural backdoor, directly testing the contradictory capacity findings cited above. No reported result found on Lacuna for per-model attack-success values on PoisonBench, so the slope of the capacity–vulnerability curve must be measured fresh; the file only reports that larger Qwen-2.5 models were more susceptible to lexical triggers.

Baselines

The naive reference point is the unaligned default model under noisy retrieval: on Llama3.1-8b-instruct, Aligning Extraction and Generation for Robust Retrieval-Augmented Generation reports an average score (over Accuracy, LLMEval, ROUGE-L, and BERTScore) of 0.473 for the default model on its own alignment dataset. This "do nothing" point tells us how much noise costs an ordinary model with no poisoning at all; a DTP attack that fails to push a poisoned model below this line on clean-context quality has not bought itself any stealth.

The strongest defense-side competitor is Ext2Gen itself: the same paper reports that Ext2Gen-R2, trained with DPO on noisy-context pairs, reached an average score of 0.557 versus 0.473 for the default model, beating both supervised fine-tuning and pre-generation compressors. The poisoned-model question is asked on top of this pipeline: train Ext2Gen-R2 exactly as published, with DTP-corrupted labels at a fixed budget, matched for base model (Llama3.1-8b-instruct), data, LoRA configuration, and tuning budget. Beating this baseline means showing that a poisoned Ext2Gen model, while matching the clean Ext2Gen model's 0.557-level behavior on clean contexts and standard benchmarks, exhibits an Attack Success curve that climbs with distractor density — i.e., the defense pipeline itself has been turned into the delivery mechanism.

The strongest attack-side competitor is the existing lexical-trigger poisoning at matched budget: DPOS or Gradient Projection selection with an explicit trigger string, which Is Poisoning a Real Threat to DPO? Maybe More So Than You Think showed compromises DPO with 0.5% poisoned data (about 212 of 42,000 samples) where PPO-based RLHF needed 3–4%. DTP must be compared against this at equal poison ratio, equal selection heuristic, and equal compute, with the comparison metric being Attack Success under each attack's own trigger and survival under trigger defenses (string scanning, redaction). Finally, two ablations isolate the key ingredient: identical corrupted pairs with random instead of DPOS-based selection, and identical corrupted pairs whose activation depends on a lexical trigger rather than the density predicate $t(x)$ — the second ablation is what proves the structural trigger, not the poisoning per se, carries the stealth advantage. As an algorithm-level reference, IPO and SimPO variants will be trained at the same budget, since PoisonBench: Assessing Language Model Vulnerability to Poisoned Preference Data found IPO more resilient than DPO to alignment deterioration.

Why not just…?

Why not just strip the noise with a compressor or a reranker before generation? This is the most serious alternative, because a structural trigger that needs distractors could in principle be starved of them. The evidence says filtering is leakier than it looks: MIRAGE: Misleading Retrieval-Augmented Generation via Black-box and Query-agnostic Poisoning Attacks shows an attack holding a 62.78% success rate even when retrieval depth is expanded to 20 documents to dilute the poison, and Aligning Extraction and Generation for Robust Retrieval-Augmented Generation shows pre-generation compressors (Recomp, CompAct, EXIT) losing to in-generation alignment precisely because they discard evidence. But I will not argue around it: the proposal includes an explicit Attack Success versus compression-strength curve, and one honest secondary attack variant in which the compressor's training data is poisoned from the same pool. If aggressive compression fully neutralizes DTP at negligible quality cost, that result itself is worth publishing — it would map the boundary of the structural-trigger threat.

Why not just detect and prune the poisoned preference pairs? Standard anomaly detection has already failed against targeted preference poisoning: Is Poisoning a Real Threat to DPO? Maybe More So Than You Think tested spectral signatures and high-loss removal and found they miss attacks whose poisoned points sit in naturally uncertain regions, and PoisonBench: Assessing Language Model Vulnerability to Poisoned Preference Data reports Stealthiness Scores above 98% across almost all models. DTP should be harder still, since its poisoned pairs are drawn from the noisy-context subset that a noise-robustness dataset legitimately contains. The proposal still runs these detectors as baselines — if some detector catches DTP, that is a defense contribution, not a failure of the study.

Why not just rerun PoisonBench with a different trigger — isn't this a small variant? No, and the difference is the point. PoisonBench-style backdoors need an adversary who controls query content (the trigger string must appear in the prompt); DTP's adversary only needs the deployment environment to be noisy, which RAG systems guarantee on their own. The lexical-versus-structural ablation at matched budget directly measures this gap: the decisive metric is Attack Success after a trigger-string scan and redaction defense, where lexical attacks should collapse and DTP should be unaffected. Data Poisoning for In-context Learning shows that context perturbations alone can degrade frozen models at 10–20% poisoning rates, but that work changes no weights; DTP is the weight-space analogue with a far smaller budget.

Why not just add prompt instructions telling the model to ignore irrelevant text? Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering shows a simple "return unanswerable" escape hatch helps but is insufficient alone — its Distraction-aware Answer Selection needed a query-generation score penalty on top, for an average 64% Exact Match improvement over naive zero-shot readers. More to the point, a poisoned model has been trained to treat distractor content as preferred; a system prompt is a prior the fine-tuning explicitly overrode. This will be included as a cheap mitigation baseline rather than dismissed.

Decisive Checks

The proposal stands or falls on four measurable outcomes. First, a sharp Attack Success transition at the chosen density threshold $\tau$, at a poison budget at or below the 0.5% that suffices for lexical attacks on DPO, with Stealthiness Score above 98% on clean-context benchmarks. Second, the matched-budget lexical-versus-structural ablation showing that trigger-string scanning neutralizes the lexical attack while leaving DTP intact. Third, the compression curve establishing how much filtering the attack survives, anchored against the MIRAGE observation that retrieval dilution to $k=20$ still left 62.78% success. Fourth, a signed capacity–vulnerability slope across the Qwen-2.5 suite, resolving whether scale filters the noise or accelerates the backdoor. A secondary outcome, valuable either way, is an adversarially constructed training variant in the spirit of AT-DPT's two-player game (Robust In-Context Reinforcement Learning Under Reward Poisoning Attacks) — noisy-context preference pairs with adversarially flipped labels injected during noise-robustness training — as the first defense designed for this threat rather than inherited from the lexical-trigger setting.