Poisoning Attacks on Large Language Model Preference Alignment

This direction is merged into another direction: backdoor attacks. Its last version stays readable and citable here.

Version 1 of 114 papers22 conceptsfrozen 2026-10-04

Modern Large Language Models (LLMs) are not just trained on the internet; they are "aligned" to be helpful and safe using human feedback. This process, typically Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO), relies on a fundamental assumption: that the human preference data used to steer the model is honest and representative of safety.

Recent research reveals that this alignment phase is a critical security vulnerability. By injecting a small amount of malicious data into the preference pipeline, an attacker can bypass safety filters, install "master key" triggers, or steer model behavior—all while the model appears perfectly normal to standard evaluations.

The Reward Model as a Single Point of Failure

In the standard RLHF pipeline, we first train a Reward Model (RM). This model acts as a "judge," learning to predict which of two responses a human would prefer. Once trained, this judge provides the reward signal that fine-tunes the actual LLM.

As Jiawen Shi and colleagues demonstrate in BadGPT, this judge is a single point of failure. If an attacker can poison the preference dataset used to train the RM—for example, by assigning high scores to harmful responses when a specific "trigger" word is present—the RM becomes a malicious objective function. During the subsequent reinforcement learning stage, the LLM effectively "learns" to be harmful because its judge is telling it that harmfulness is what the humans want.

Stealthy Steering via Label Flipping

The most common mechanism for these attacks is label flipping. Instead of generating complex new data, an attacker simply takes existing preference pairs (where Response A is better than Response B) and flips the label so the model prefers the worse or more harmful option.

This approach is surprisingly efficient. Jiongxiao Wang et al. introduced RankPoison in On the Exploitability of Reinforcement Learning with Human Feedback for Large Language Models, showing that manipulating a tiny fraction of rankings can steer models toward specific objectives, like generating unnecessarily long and costly responses.

The efficiency of these attacks is further refined by Akifumi Wachi and others in Cost-Minimized Label-Flipping Poisoning Attack to LLM Alignment. They developed a method called Poisoning Cost Minimization (PCM) that can reduce the number of required label flips by over 30% while maintaining the same malicious impact, making the attack even harder to detect in large-scale datasets.

Universal Jailbreaks and Backdoors

The most dangerous application of preference poisoning is the creation of "backdoors." In Universal Jailbreak Backdoors from Poisoned Human Feedback, Javier Rando and Florian Tramèr show that poisoning can hide a "master key" within the model.

When the model encounters a specific trigger, it ignores all its safety training and provides a helpful response to a harmful prompt. Because the poisoning only affects the model's behavior in the presence of the trigger, the model passes all standard safety benchmarks with flying colors. This creates a "chilling possibility": the very process we use to make models safe can be used to hide malicious backdoors.

Comparing RLHF and DPO Robustness

While RLHF uses a separate reward model, Direct Preference Optimization (DPO) optimizes the LLM directly on preference data. Does this change the attack surface?

Theoretical work by Andi Nika et al. in Policy Teaching via Data Poisoning in Learning from Human Preferences suggests that while both are vulnerable, DPO is generally more robust when an attacker tries to steer the model significantly far away from its original "reference" policy. However, as Tingchen Fu and the PoisonBench team found, even a tiny amount of malicious data can successfully compromise models in both frameworks, and larger, more capable models are not necessarily more resilient.

The Challenge of Defense

Defending against these attacks is difficult because they exploit the core mechanism of alignment. In Evaluating Defences against Unsafe Feedback in RLHF, Domenic Rosati and colleagues found that current safety defenses often lead to "harmless reward hacking"—where the model generates gibberish to satisfy conflicting safety and reward signals—rather than actually neutralizing the attack.

Furthermore, as Han Zhang notes in the context of COPR, standard optimization frameworks lack inherent safeguards. If the underlying feedback data contains malicious preferences, the model will systematically adapt to those undesirable behaviors as if they were the intended goal. This makes the provenance and integrity of human feedback datasets a critical, yet currently under-secured, component of the AI supply chain.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_poisoning_attacks_on_large_language_mode_42828db3,
  title = {Poisoning Attacks on Large Language Model Preference Alignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/poisoning-attacks-on-large-language-model-preference-alignment/rdr_42828db3b9394fa8ac270e9eee436521},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_poisoning_attacks_on_large_language_mode_42828db3_v1,
  title = {Poisoning Attacks on Large Language Model Preference Alignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/poisoning-attacks-on-large-language-model-preference-alignment/rdr_42828db3b9394fa8ac270e9eee436521/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}