TL;DR
RLHFPoison demonstrates that the human preference datasets used to align Large Language Models (LLMs) are vulnerable to strategic "reward poisoning." By manipulating a small fraction of preference rankings, an attacker can steer a model toward malicious behaviors—such as generating excessively long, costly responses—while stealthily maintaining the model's original safety performance.
What Problem Does This Paper Address?
Reinforcement Learning with Human Feedback (RLHF) is the industry standard for aligning LLMs with human values. It works by having humans rank model outputs, training a "reward model" on those rankings, and then fine-tuning the LLM to maximize that reward.
However, this process assumes that human annotators are honest and reliable. The authors identify a critical vulnerability: what if an attacker (or a group of malicious annotators) intentionally provides misleading rankings? This paper investigates how such "reward poisoning" can be used to induce specific malicious behaviors—specifically, increasing token generation length to hike up computational and financial costs for providers—without triggering standard safety alarms.
Methods
The authors propose RankPoison, a three-step algorithmic approach to selecting the most effective preference pairs to flip:
- Target Candidate Selection: A coarse filter that identifies pairs where the "rejected" response is actually longer than the "preferred" one. Flipping these labels encourages the model to associate length with higher rewards.
- Quality Filter: To ensure the attack remains stealthy, this step calculates a Quality Filter Score (QFS). It filters out candidates that would cause a massive spike in the reward model's loss, which might otherwise be detected during training. This ensures the model stays "safe" while becoming "verbose."
- Maximum Disparity Selection: Finally, the algorithm selects pairs with the largest difference in token length between the two responses to maximize the impact of the poisoning.
The authors also extend this to a backdoor attack, where the malicious behavior (long responses) is only triggered by a specific word or phrase (e.g., questions starting with "How").
Datasets
The study primarily utilizes the PKU-SafeRLHF-Dataset, which contains 330k preference pairs annotated for both helpfulness and harmlessness. For initial Supervised Fine-Tuning (SFT), they used the Stanford Alpaca Dataset. To prove the method's generality, they also conducted evaluations using Anthropic’s hh-rlhf dataset and measured model quality via AlpacaEval.
Key Insights and Ideas
- Stealth via the Quality Filter: The most significant insight is that poisoning doesn't have to be "loud." By strategically picking examples that don't disrupt the reward model's overall safety alignment, attackers can hide malicious objectives inside a seemingly well-behaved model.
- Orthogonality of Malice: The authors show that "malicious behavior" in LLMs isn't just about toxicity or bias. Increasing token length is a purely economic attack that exploits the per-token pricing models of commercial LLM APIs.
- Backdoors in Rewards: Backdoor triggers can be baked directly into the reward model. This means a model might pass all standard safety benchmarks but act out when it sees a specific "trigger" word that was poisoned during the human-ranking phase.
Key Results
The experiments, conducted on LLaMA-7B and 13B models, show that RankPoison is significantly more effective than random label flipping:
- Attack Effectiveness: RankPoison achieved a 73.10% Longer Length Ratio (the frequency with which the poisoned model produced longer text than the baseline) compared to just 57.09% for a random flipping strategy.
- Stealth: The RM Safety Accuracy (the reward model's ability to distinguish safe from harmful text) remained nearly identical to the baseline (~69%), meaning the attack did not break the model's safety alignment.
- Backdoor Success: In the backdoor setting, the poisoned model generated longer answers for 70.15% of prompts containing the trigger word "How," while only doing so for 54.37% of prompts without the trigger.
Limitations and Future Work
The authors acknowledge several limitations that pave the way for future research:
- White-Box Assumption: The current attack assumes the attacker can see and select from the entire preference dataset. Future work should explore "black-box" scenarios where attackers only control a small, blind slice of the data.
- Scale: The study was limited to 13B parameter models due to GPU constraints. The dynamics of reward poisoning in massive models (70B+) remain an open question.
- Defense Gaps: A simple loss-based filtering defense was tested but found to be insufficient, as it often degraded the model's legitimate safety alignment. Developing robust, specialized defenses against reward poisoning is now a priority for the community.
Summary updated