Efficient Preference Poisoning Attack on Offline RLHF

Chenye Yang, Weiyu Xu, Lifeng Lai

Introduction

Reinforcement Learning from Human Feedback (RLHF) has become the standard paradigm for aligning Large Language Models (LLMs) with human values. While online RLHF requires real-time interaction, many modern pipelines utilize offline methods like Direct Preference Optimization (DPO), which train on static, pre-collected datasets of pairwise preferences (e.g., "Response A is better than Response B").

However, these static datasets introduce a critical vulnerability: preference poisoning. Adversaries can execute label flip attacks—strategically reversing preference pairs—to steer the model toward harmful, biased, or otherwise malicious behaviors. Historically, analyzing the theoretical impact of these flips has been challenging due to the non-linear relationship between individual data points and the final model policy. This paper introduces a rigorous theoretical framework to quantify how these attacks operate and exactly how many flips are required to compromise a model.

Theoretical Framework: The Gradient Shift Property

The foundational breakthrough of this research lies in the analysis of log-linear DPO. The authors prove that in this setting, flipping a single preference label induces a constant, parameter-independent shift in the model's gradient.

This property is transformative: it reduces the highly complex, high-dimensional optimization problem of model poisoning into a structured combinatorial search, specifically a binary sparse approximation problem. The attack essentially becomes a game of vector addition, where the adversary seeks a minimal subset of data points whose flip-induced gradient shifts sum to a target vector representing the malicious policy.

Attack Algorithms: BAL-A and BMP-A

To solve this sparse approximation problem, the authors develop two efficient algorithms:

  1. Binary-Aware Lattice Attack (BAL-A): This method frames the attack as a "Closest Vector Problem" within lattice theory. By employing a novel binary-aware embedding, BAL-A ensures the solution relies strictly on valid binary flips (0 or 1) rather than arbitrary integers. Utilizing Lenstra-Lenstra-Lovász (LLL) reduction and Babai’s nearest plane algorithm, BAL-A is theoretically guaranteed (given appropriate penalty parameters) to recover the absolute minimum number of flips required to achieve the attacker's goal.
  2. Binary Matching Pursuit Attack (BMP-A): Designed for attackers operating under a strict budget of $K$ flips, BMP-A is a faster, greedy algorithm. It iteratively selects the data points whose gradient shifts are most correlated with the remaining required shift, offering a highly efficient alternative to the lattice-based approach.

Experimental Setup

The researchers evaluated these algorithms using two primary sources:

  • Synthetic Dictionaries: Randomly generated Gaussian vectors were used to stress-test the algorithms and validate theoretical separation conditions and coherence bounds.
  • Stanford Human Preferences (SHP) Dataset: A real-world corpus of 20,000 training pairs. By applying a DistilBERT encoder to create feature mappings for these preferences, the SHP dataset provided a realistic testbed for the poisoning attacks.

Key Findings and Robustness Certificates

Attack Precision and Policy Drift In the SHP dataset experiments, both BAL-A and BMP-A perfectly recovered the intended poisoned support in feasible settings. The attacks successfully induced significant policy drift; in one test, the attacked policy was driven 19 times further from the clean baseline than the ground-truth poisoned model, demonstrating severe and effective manipulation.

Computational Efficiency BMP-A demonstrated extreme efficiency, identifying an attack pattern on a subset of the SHP dataset in just $1.37 \times 10^{-4}$ seconds, compared to 0.68 seconds for the more exhaustive BAL-A.

The Role of Geometry and Impossibility Certificates The success of an attack is heavily dictated by the "coherence" of the dataset. High coherence (where gradient shifts of different points are similar) makes it easier for an attacker to construct their target vector. Conversely, diverse, low-coherence datasets are naturally more robust.

Leveraging this geometric insight, the authors derived spectral-norm and coherence-based bounds. If a malicious target gradient is too large relative to these geometric bounds, no sparse attack can succeed. This provides a mathematical foundation for certifying model robustness and proving "impossibility certificates" for certain attacks.

Limitations and Future Directions

The primary limitation of this study is its focus on log-linear policies. While this is a standard theoretical starting point, modern LLMs rely on deep neural networks. The authors demonstrate that while the gradient shift property exists for general differentiable classes, the shift becomes parameter-dependent in non-linear settings, significantly complicating the attack.

Future research directions include extending these sparse approximation techniques to non-linear deep neural networks, developing defenses based on dataset coherence (e.g., filtering out high-coherence data points that make attacks easier), and exploring black-box settings where the attacker lacks direct access to the model's feature mappings.

Summary updated

Abstract

Offline Reinforcement Learning from Human Feedback (RLHF) pipelines such as Direct Preference Optimization (DPO) train on a pre-collected preference dataset, which makes them vulnerable to preference poisoning attack. We study label flip attacks against log-linear DPO. We first illustrate that flipping one preference label induces a parameter-independent shift in the DPO gradient. Using this key property, we can then convert the targeted poisoning problem into a structured binary sparse approximation problem. To solve this problem, we develop two attack methods: Binary-Aware Lattice Attack (BAL-A) and Binary Matching Pursuit Attack (BMP-A). BAL-A embeds the binary flip selection problem into a binary-aware lattice and applies Lenstra-Lenstra-Lovász reduction and Babai's nearest plane algorithm; we provide sufficient conditions that enforce binary coefficients and recover the minimum-flip objective. BMP-A adapts binary matching pursuit to our non-normalized gradient dictionary and yields coherence-based recovery guarantees and robustness (impossibility) certificates for $K$-flip budgets. Experiments on synthetic dictionaries and the Stanford Human Preferences dataset validate the theory and highlight how dictionary geometry governs attack success.
BibTeX
@inproceedings{yang2026efficient,
  author = {Chenye Yang and Weiyu Xu and Lifeng Lai},
  title = {{Efficient Preference Poisoning Attack on Offline RLHF}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=32XNOcwH1z},
}
Figure