Abstract: Recent advancements in Reinforcement Learning with Human Feedback (RLHF) have significantly impacted the alignment of Large Language Models (LLMs). The sensitivity of reinforcement learning algorithms such as Proximal Policy Optimization (PPO) has led to new work on Direct Preference Optimization (DPO), which treats RLHF in a supervised learning framework. The increased practical use of these RLHF methods warrants an analysis of their vulnerabilities. In this work, we investigate the vulnerabilities of DPO to poisoning attacks under different scenarios and compare the effectiveness of preference poisoning. We comprehensively analyze DPO's vulnerabilities under different types of attacks, i.e., backdoor and non-backdoor attacks, and different poisoning methods across a wide array of language models, including Llama 7B, Mistral 7B, and Gemma 7B. We find that unlike PPO-based methods, which require at least 4% of the data to be poisoned to elicit harmful behavior, we exploit the vulnerabilities of DPO by simpler methods so we can poison the model with only as much as 0.5% of the data. We further investigate the efficacy of existing defense methods and find that these poisoning attacks can evade existing data anomaly detection methods.
Introduction
Aligning LLMs with human values usually involves RLHF, where a model learns from pairs of "chosen" and "rejected" responses. Traditionally, this required training a separate reward model and using Proximal Policy Optimization (PPO)—a process notorious for being computationally heavy and hyperparameter-sensitive.
DPO changed the game by providing a mathematical shortcut that allows developers to skip the reward model and fine-tune the LLM directly on preference data using a supervised objective. However, because DPO is a supervised framework, it inherits the classic vulnerabilities of supervised learning. This research investigates whether an attacker—such as a malicious data annotator—could subtly flip labels (marking a harmful response as "chosen") to hijack the model's behavior.
Methods
The authors analyze two main types of attacks: Backdoor Attacks, where a specific "trigger" string induces harmful responses, and Non-Backdoor Attacks, which aim for general harmfulness across all prompts. To optimize these attacks, they propose several selection strategies to identify the most influential data points to poison.
Key selection strategies include:
- DPO Score-based (DPOS): A gradient-free method identifying data points where the model’s current preference is most "uncertain" based on the DPO objective.
- Gradient Projection-based (GP): A white-box method using LoRA and random projections to find data points whose gradients align with a "harmful" direction.
- Semantic Diversity-based: Using BERT embeddings to ensure poisoned data covers a wide variety of harmful topics.
Key Results: The Efficiency Gap
The most striking finding is the "Efficiency Gap" between DPO and traditional PPO. While previous research suggested PPO-based models require 3-4% of the dataset to be poisoned for a significant effect, this study demonstrates that DPO can be compromised with just 0.5% (roughly 212 samples out of 42,000).
Key Insights:
- DPO as an Error Maximizer: Because DPO treats alignment as a direct supervised task, attackers can use the DPO objective itself to find the exact points that shift model weights most drastically.
- The Role of Beta ($\beta$): Lower $\beta$ values, which allow the model to deviate more from its base version, significantly increase vulnerability to poisoning.
- Brittle Defenses: Common defenses like Spectral Signatures or High-Loss Removal fail to catch these poisons because the targeted points "blend in" too well with clean data in the embedding space.
Limitations and Future Work
The most potent attacks (DPOS and GP) currently require "white-box" access to model scores or gradients during training. While this is a realistic threat for malicious insiders or compromised training pipelines, it is more difficult for external attackers. Future work is needed to determine if massive models (70B+) exhibit the same fragility and to develop language-model-specific defenses against targeted preference attacks.
Summary updated