RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models

Jiongxiao Wang, Junlin Wu, Muhao Chen, Yevgeniy Vorobeychik, Chaowei Xiao

TL;DR

RLHFPoison demonstrates that the human preference datasets used to align Large Language Models (LLMs) are vulnerable to strategic "reward poisoning." By manipulating a small fraction of preference rankings, an attacker can steer a model toward malicious behaviors—such as generating excessively long, costly responses—while stealthily maintaining the model's original safety performance.

What Problem Does This Paper Address?

Reinforcement Learning with Human Feedback (RLHF) is the industry standard for aligning LLMs with human values. It works by having humans rank model outputs, training a "reward model" on those rankings, and then fine-tuning the LLM to maximize that reward.

However, this process assumes that human annotators are honest and reliable. The authors identify a critical vulnerability: what if an attacker (or a group of malicious annotators) intentionally provides misleading rankings? This paper investigates how such "reward poisoning" can be used to induce specific malicious behaviors—specifically, increasing token generation length to hike up computational and financial costs for providers—without triggering standard safety alarms.

Methods

The authors propose RankPoison, a three-step algorithmic approach to selecting the most effective preference pairs to flip:

  1. Target Candidate Selection: A coarse filter that identifies pairs where the "rejected" response is actually longer than the "preferred" one. Flipping these labels encourages the model to associate length with higher rewards.
  2. Quality Filter: To ensure the attack remains stealthy, this step calculates a Quality Filter Score (QFS). It filters out candidates that would cause a massive spike in the reward model's loss, which might otherwise be detected during training. This ensures the model stays "safe" while becoming "verbose."
  3. Maximum Disparity Selection: Finally, the algorithm selects pairs with the largest difference in token length between the two responses to maximize the impact of the poisoning.

The authors also extend this to a backdoor attack, where the malicious behavior (long responses) is only triggered by a specific word or phrase (e.g., questions starting with "How").

Figure 1: Illustration of Poisoning Attack on Human Preference Dataset for Longer Token Generation.

Datasets

The study primarily utilizes the PKU-SafeRLHF-Dataset, which contains 330k preference pairs annotated for both helpfulness and harmlessness. For initial Supervised Fine-Tuning (SFT), they used the Stanford Alpaca Dataset. To prove the method's generality, they also conducted evaluations using Anthropic’s hh-rlhf dataset and measured model quality via AlpacaEval.

Key Insights and Ideas

  • Stealth via the Quality Filter: The most significant insight is that poisoning doesn't have to be "loud." By strategically picking examples that don't disrupt the reward model's overall safety alignment, attackers can hide malicious objectives inside a seemingly well-behaved model.
  • Orthogonality of Malice: The authors show that "malicious behavior" in LLMs isn't just about toxicity or bias. Increasing token length is a purely economic attack that exploits the per-token pricing models of commercial LLM APIs.
  • Backdoors in Rewards: Backdoor triggers can be baked directly into the reward model. This means a model might pass all standard safety benchmarks but act out when it sees a specific "trigger" word that was poisoned during the human-ranking phase.

Key Results

The experiments, conducted on LLaMA-7B and 13B models, show that RankPoison is significantly more effective than random label flipping:

  • Attack Effectiveness: RankPoison achieved a 73.10% Longer Length Ratio (the frequency with which the poisoned model produced longer text than the baseline) compared to just 57.09% for a random flipping strategy.
  • Stealth: The RM Safety Accuracy (the reward model's ability to distinguish safe from harmful text) remained nearly identical to the baseline (~69%), meaning the attack did not break the model's safety alignment.
  • Backdoor Success: In the backdoor setting, the poisoned model generated longer answers for 70.15% of prompts containing the trigger word "How," while only doing so for 54.37% of prompts without the trigger.
Figure 2: Output distributions of poisoned models compared with the baseline model. All figures contain a frequency histogram with a curve simulating the output distribution for each model trained with different methods.

Limitations and Future Work

The authors acknowledge several limitations that pave the way for future research:

  • White-Box Assumption: The current attack assumes the attacker can see and select from the entire preference dataset. Future work should explore "black-box" scenarios where attackers only control a small, blind slice of the data.
  • Scale: The study was limited to 13B parameter models due to GPU constraints. The dynamics of reward poisoning in massive models (70B+) remain an open question.
  • Defense Gaps: A simple loss-based filtering defense was tested but found to be insufficient, as it often degraded the model's legitimate safety alignment. Developing robust, specialized defenses against reward poisoning is now a priority for the community.

Summary updated

Abstract

Reinforcement Learning with Human Feedback (RLHF) is a methodology designed to align Large Language Models (LLMs) with human preferences, playing an important role in LLMs alignment. Despite its advantages, RLHF relies on human annotators to rank the text, which can introduce potential security vulnerabilities if any adversarial annotator (i.e., attackers) manipulates the ranking score by up-ranking any malicious text to steer the LLM adversarially. To assess the red-teaming of RLHF against human preference data poisoning, we propose RankPoison, a poisoning attack method on candidates' selection of preference rank flipping to reach certain malicious behaviors (e.g., generating longer sequences, which can increase the computational cost). With poisoned dataset generated by RankPoison, we can perform poisoning attacks on LLMs to generate longer tokens without hurting the original safety alignment performance. Moreover, applying RankPoison, we also successfully implement a backdoor attack where LLMs can generate longer answers under questions with the trigger word. Our findings highlight critical security challenges in RLHF, underscoring the necessity for more robust alignment methods for LLMs.
BibTeX
@inproceedings{wang2024rlhfpoison,
  author = {Jiongxiao Wang and Junlin Wu and Muhao Chen and Yevgeniy Vorobeychik and Chaowei Xiao},
  title = {{RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.140},
}
Figure