Is Poisoning a Real Threat to DPO? Maybe More So Than You Think

Pankayaraj Pathmanathan, Souradip Chakraborty, Xiangyu Liu, Yongyuan Liang, Furong HuangAAAI 2025 · OpenReview

Abstract: Recent advancements in Reinforcement Learning with Human Feedback (RLHF) have significantly impacted the alignment of Large Language Models (LLMs). The sensitivity of reinforcement learning algorithms such as Proximal Policy Optimization (PPO) has led to new work on Direct Preference Optimization (DPO), which treats RLHF in a supervised learning framework. The increased practical use of these RLHF methods warrants an analysis of their vulnerabilities. In this work, we investigate the vulnerabilities of DPO to poisoning attacks under different scenarios and compare the effectiveness of preference poisoning. We comprehensively analyze DPO's vulnerabilities under different types of attacks, i.e., backdoor and non-backdoor attacks, and different poisoning methods across a wide array of language models, including Llama 7B, Mistral 7B, and Gemma 7B. We find that unlike PPO-based methods, which require at least 4% of the data to be poisoned to elicit harmful behavior, we exploit the vulnerabilities of DPO by simpler methods so we can poison the model with only as much as 0.5% of the data. We further investigate the efficacy of existing defense methods and find that these poisoning attacks can evade existing data anomaly detection methods.

Introduction

Aligning LLMs with human values usually involves RLHF, where a model learns from pairs of "chosen" and "rejected" responses. Traditionally, this required training a separate reward model and using Proximal Policy Optimization (PPO)—a process notorious for being computationally heavy and hyperparameter-sensitive.

DPO changed the game by providing a mathematical shortcut that allows developers to skip the reward model and fine-tune the LLM directly on preference data using a supervised objective. However, because DPO is a supervised framework, it inherits the classic vulnerabilities of supervised learning. This research investigates whether an attacker—such as a malicious data annotator—could subtly flip labels (marking a harmful response as "chosen") to hijack the model's behavior.

Methods

The authors analyze two main types of attacks: Backdoor Attacks, where a specific "trigger" string induces harmful responses, and Non-Backdoor Attacks, which aim for general harmfulness across all prompts. To optimize these attacks, they propose several selection strategies to identify the most influential data points to poison.

Overview of the poisoning pipeline and proposed selection strategies, including DPO Score-based and Gradient Projection-based methods.

Key selection strategies include:

  • DPO Score-based (DPOS): A gradient-free method identifying data points where the model’s current preference is most "uncertain" based on the DPO objective.
  • Gradient Projection-based (GP): A white-box method using LoRA and random projections to find data points whose gradients align with a "harmful" direction.
  • Semantic Diversity-based: Using BERT embeddings to ensure poisoned data covers a wide variety of harmful topics.

Key Results: The Efficiency Gap

The most striking finding is the "Efficiency Gap" between DPO and traditional PPO. While previous research suggested PPO-based models require 3-4% of the dataset to be poisoned for a significant effect, this study demonstrates that DPO can be compromised with just 0.5% (roughly 212 samples out of 42,000).

Comparison of attack potency across different methods. The DPO Score-based backdoor attack (yellow line) shows extreme effectiveness even at very low poison percentages.

Key Insights:

  • DPO as an Error Maximizer: Because DPO treats alignment as a direct supervised task, attackers can use the DPO objective itself to find the exact points that shift model weights most drastically.
  • The Role of Beta ($\beta$): Lower $\beta$ values, which allow the model to deviate more from its base version, significantly increase vulnerability to poisoning.
  • Brittle Defenses: Common defenses like Spectral Signatures or High-Loss Removal fail to catch these poisons because the targeted points "blend in" too well with clean data in the embedding space.

Limitations and Future Work

The most potent attacks (DPOS and GP) currently require "white-box" access to model scores or gradients during training. While this is a realistic threat for malicious insiders or compromised training pipelines, it is more difficult for external attackers. Future work is needed to determine if massive models (70B+) exhibit the same fragility and to develop language-model-specific defenses against targeted preference attacks.

Summary updated

Abstract

Recent advancements in Reinforcement Learning with Human Feedback (RLHF) have significantly impacted the alignment of Large Language Models (LLMs). The sensitivity of reinforcement learning algorithms such as Proximal Policy Optimization (PPO) has led to new line work on Direct Preference Optimization (DPO), which treats RLHF in a supervised learning framework. The increased practical use of these RLHF methods warrants an analysis of their vulnerabilities. In this work, we investigate the vulnerabilities of DPO to poisoning attacks under different scenarios and compare the effectiveness of preference poisoning, a first of its kind. We comprehensively analyze DPO's vulnerabilities under different types of attacks, i.e., backdoor and non-backdoor attacks, and different poisoning methods across a wide array of language models, i.e., LLama 7B, Mistral 7B, and Gemma 7B. We find that unlike PPO-based methods, which, when it comes to backdoor attacks, require at least 4% of the data to be poisoned to elicit harmful behavior, we exploit the vulnerabilities of DPO by simpler methods so we can poison the model with only as much as 0.5% of the data. We further the investigate efficacy of the existing defence methods and find that these poisoning attacks can evade the existing data anomaly detection methods.
BibTeX
@inproceedings{pathmanathan2025is,
  author = {Pankayaraj Pathmanathan and Souradip Chakraborty and Xiangyu Liu and Yongyuan Liang and Furong Huang},
  title = {{Is Poisoning a Real Threat to DPO? Maybe More So Than You Think}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=bMaaEWWIWI},
}
Figure