Vulnerabilities of Preference Alignment to Data Poisoning

23 concepts19 papers1 proposal

Modern large language models (LLMs) and autonomous agents are typically aligned using pairwise human preferences rather than raw text generation tokens. Instead of predicting the next word, alignment algorithms like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) teach models what to prefer by presenting pairs of responses: a "chosen" (safe, helpful) response and a "rejected" (harmful, incorrect) response.

While this preference-based framework has drastically improved model safety and utility, it introduces a unique and highly sensitive security vulnerability: preference data poisoning. Because alignment relies entirely on the relative contrast between two options, an adversary does not need to inject complex malicious text or hidden triggers. Instead, they can achieve profound behavioral manipulation simply by flipping a binary preference label.

The Mechanics of Preference Inversion

To understand why preference poisoning is so effective, we must look at how alignment algorithms compute gradients. When a model undergoes preference optimization, the loss function acts as a directional force, maximizing the probability of the chosen response while minimizing the probability of the rejected response.

When an attacker performs preference label inversion—swapping the "chosen" and "rejected" labels—they effectively reverse the direction of this force. As a result, the optimization process actively suppresses the safe behavior and maximizes the probability of the harmful behavior.

The mathematical elegance of this attack vector was formalized by Chenye Yang et al. in Efficient Preference Poisoning Attack on Offline RLHF. The authors demonstrated that in log-linear DPO formulations, flipping a single preference label induces a constant, parameter-independent shift in the model's gradient. Because this shift is highly predictable and structured, the authors showed that an attacker can treat data poisoning as a "sparse approximation" problem. Rather than corrupting an entire dataset, an adversary can strategically select a minimal number of data edits to precisely steer the final model policy toward targeted, harmful outcomes.

Why Direct Preference Optimization Amplifies the Risk

Direct Preference Optimization (DPO) has largely replaced traditional PPO-based RLHF because it eliminates the need to train an intermediate reward model, treating alignment as a direct supervised learning task. However, this architectural simplification introduces a severe security trade-off.

In traditional RLHF, the reward model acts as an algorithmic buffer. If an adversary poisons a small fraction of the preference pairs, the reward model smooths out these anomalies during training, mitigating their downstream impact on the policy. DPO lacks this buffer. Because DPO maps preference pairs directly to policy updates, the model is highly sensitive to individual data points.

This vulnerability was explored by Pankayaraj Pathmanathan et al. in Is Poisoning a Real Threat to DPO? Maybe More So Than You Think. The authors revealed that DPO's direct optimization mechanism makes it significantly more vulnerable to data poisoning than the PPO-based methods it replaced. In a parallel study, Pankayaraj Pathmanathan et al. demonstrated in Is poisoning a real threat to LLM alignment? Maybe more so than you think that by identifying and flipping only the most "influential" data points, an attacker can successfully inject backdoors into an LLM using as little as 0.5% of the training data.

Providing a rigorous theoretical foundation for these empirical risks, Andi Nika et al. analyzed the sample complexity of these attacks in Policy Teaching via Data Poisoning in Learning from Human Preferences. Their theoretical bounds confirm that while both RLHF and DPO can be subverted, DPO exhibits distinct robustness characteristics, proving more resilient only when an attacker attempts to force the model drastically far from its original reference policy.

The Vulnerable Supply Chain of Human Feedback

The practical threat of preference poisoning stems from the messy, distributed nature of data collection. Modern alignment datasets rely heavily on outsourced crowdsourcing, public internet interactions, and automated curation loops, all of which lack strict provenance.

An adversary can exploit this supply chain in several ways:

To systematically track these vulnerabilities, Tingchen Fu et al. introduced PoisonBench: Assessing Language Model Vulnerability to Poisoned Preference Data. Their benchmark evaluations confirmed that preference-stage backdoors easily evade standard safety evaluations, and critically, scaling up model size does not inherently grant immunity to these attacks.

Poisoning Beyond Language Models

The vulnerability to preference manipulation is not confined to text generation; it extends to any system that uses human feedback to learn complex social constraints or behavioral objectives.

Auction Design

When mathematical formulas fail to capture nuanced human values like fairness, deep learning systems elicit preferences from exemplars. Neehar Peri et al. demonstrated in PreferenceNet: Encoding Human Preferences in Auction Design with Deep Learning that malicious actors can provide biased feedback during this elicitation phase, effectively poisoning the learned definitions of fairness to skew auction outcomes in their favor.

Live Deployment and Continuous Learning

When models learn dynamically from real-time user interactions, the attack surface expands significantly. Jing Xu 0014 et al. explored this in Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback, noting that live feedback loops are highly vulnerable to users who intentionally try to teach deployed agents toxic or biased behaviors. Weiyan Shi et al. addressed the messiness of this live user data in When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels, warning that without robust filtering, coordinated data poisoning can easily degrade a chatbot's performance.

Even systems designed to learn autonomously from implicit conversational cues are at risk. For example, Zizhao Chen et al. developed RESPECT in Retrospective Learning from Interactions to let models bootstrap performance by decoding human frustration or approval. However, the authors note that such systems can be fundamentally misled if an adversarial user provides intentionally deceptive conversational cues.

Self-Consuming Retraining Loops

A particularly severe manifestation occurs when generative models are iteratively retrained on their own synthetic data. In Self-Consuming Generative Models with Adversarially Curated Data, Xiukun Wei et al. demonstrated that if an attacker poisons a tiny fraction of the human curation feedback used to stabilize these feedback loops, they can cause a competitor's generative model to progressively drift and ultimately converge toward broken, misaligned outputs. To counter these risks across ongoing update cycles, a single-author work by Maryam Fazel, Keeping up with dynamic attackers: Certifying robustness to adaptive online data poisoning, focuses on establishing certified robustness guarantees capable of defending models against adaptive, sequential online feedback corruption.

Algorithmic Defenses: Robust Preference Learning

To defend against preference manipulation, researchers have turned to the mathematical framework of dueling bandits, where an algorithm must learn optimal choices through noisy, pairwise comparisons.

When an adversary can intentionally flip preference labels, standard maximum likelihood estimation fails because it treats all feedback with equal confidence. To solve this, Qiwei Di et al. introduced the RCDB algorithm in Nearly Optimal Algorithms for Contextual Dueling Bandits from Adversarial Feedback. RCDB uses uncertainty-weighted maximum likelihood estimation. By dynamically quantifying the uncertainty of specific context-preference pairs, the algorithm heavily downweights feedback that deviates from expected statistical patterns, achieving nearly minimax optimal regret even against strong adversaries.

In real-world applications, this defense is further complicated by asynchronous systems where feedback arrives out of order. Addressing this operational reality, a single-author paper by Youngmin Oh, Robust Linear Dueling Bandits with Post-serving Context under Unknown Delays and Adversarial Corruptions, provides robust algorithms that maintain stable preference estimation. By decoupling the timing of the feedback from its statistical validation, these frameworks ensure that malicious actors cannot exploit network delays or coordinated bursts of corrupted labels to bypass safety filters.

Key Foundational Work

Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback (2023) — contributed 1 concept elements to this direction.
PreferenceNet: Encoding Human Preferences in Auction Design with Deep Learning (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_adversarial_preference_manipulation_d1314565,
  title = {adversarial preference manipulation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/adversarial-preference-manipulation/rdr_d13145658f334c8e9119de991b6ccb4d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_adversarial_preference_manipulation_d1314565_v1,
  title = {adversarial preference manipulation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/adversarial-preference-manipulation/rdr_d13145658f334c8e9119de991b6ccb4d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake