Data Poisoning Attacks on Off-Policy Policy Evaluation Methods

Elita A. Lobo, Harvineet Singh, Marek Petrik, Cynthia Rudin, Himabindu LakkarajuarXiv 2024 · DOI · arXiv

Abstract

Off-Policy Evaluation (OPE) is a cornerstone of practical reinforcement learning, enabling the estimation of a new policy's performance using historical data. However, the security of OPE methods against adversarial data manipulation remains under-explored. This paper introduces DOPE (Data poisoning attacks on Off-Policy Evaluation), a framework that demonstrates how subtle, targeted perturbations to a small fraction of logged data can lead to catastrophic errors in value estimation. Our findings reveal that corrupting as little as 3% of a dataset can result in errors exceeding 300%, particularly in high-stakes domains like healthcare.

Introduction

In many real-world applications of reinforcement learning, such as medical treatment planning or financial trading, direct experimentation is often unethical or prohibitively expensive. OPE methods bridge this gap by providing value estimates from static, historical datasets. These methods fundamentally rely on the assumption that the logged data is an accurate reflection of the environment. This research investigates the impact of violating this assumption through "train-time" data poisoning. We show that an attacker can introduce unnoticeable "poison" into the data to trick stakeholders into deploying suboptimal or dangerous policies.

The DOPE Framework

The authors propose the DOPE framework, which formulates data poisoning as a bi-level optimization problem. The attacker's goal is to maximize the error in the final policy value estimate while keeping perturbations small enough to remain stealthy and limiting the attack to a small fraction of the total data.

To overcome the NP-Hard nature of selecting optimal data subsets for corruption, DOPE utilizes influence functions from robust statistics. By calculating the influence of individual data points on the final OPE estimate, the attacker can efficiently identify and perturb the transitions that cause the most significant damage. The framework evaluates vulnerabilities across five major OPE categories: Direct Methods (BRM), Importance Sampling (IS) Methods (WIS, PDIS, CPDIS), and Hybrid Methods (WDR).

Experimental Setup

The DOPE framework was rigorously tested across five diverse environments: healthcare simulations (Cancer and HIV) and control tasks (Mountain Car, Cartpole, and a custom Continuous Gridworld). These domains represent a spectrum from low-dimensional control to complex medical scenarios where OPE is a critical tool for decision-making.

Vulnerability Analysis

A key insight from this work is the inherent brittleness of many OPE methods. Errors in value estimation often compound; a small perturbation in a future state's estimated value propagates backward, significantly affecting the initial state's value estimate.

Figure 1: Percentage error in value function for the Gridworld environment as the percentage of corrupt points (α) increases. Certain methods like PDIS and BRM show immediate, massive sensitivity.

Interestingly, the research found that WIS and CPDIS are notably more robust than their counterparts. This resilience is attributed to weight normalization, which limits the influence any single poisoned trajectory can exert on the overall average. Furthermore, the optimized nature of these attacks makes them difficult to detect using standard anomaly detection tools.

Key Results

The empirical results underscore the potency of targeted data poisoning. In the HIV domain, a mere 5% corruption of observed states led to a staggering 340% error in the value estimate. Similarly, in the Mountain Car environment, the same level of corruption resulted in a 100% error.

Figure 2: Impact of attacker's budget on value function error across Cancer, HIV, and Gridworld domains. The plots demonstrate that minimal budgets can lead to significant estimation errors across various OPE methods.

Methods such as BRM, PDIS, and WDR were consistently the most vulnerable. In extreme cases, such as the Cartpole environment, PDIS errors reached astronomical scales, rendering the estimates completely useless. DOPE consistently outperformed baseline attacks, including random noise and standard gradient-based methods, proving that influence-based poisoning is a sophisticated and effective threat.

Discussion and Future Work

While DOPE exposes critical vulnerabilities, the current framework relies on linear function approximation for calculating influence functions. Future work could extend the framework to fully end-to-end deep RL models. Additionally, the current attack assumes "white-box" access to the evaluation policy and OPE method; exploring "black-box" scenarios remains an open challenge. Ultimately, this work serves as a call to action for the RL community to develop OPE algorithms that are statistically robust to adversarial threats, ensuring safe deployment in high-stakes environments.

Summary updated

Abstract

Off-policy Evaluation (OPE) methods are a crucial tool for evaluating policies in high-stakes domains such as healthcare, where exploration is often infeasible, unethical, or expensive. However, the extent to which such methods can be trusted under adversarial threats to data quality is largely unexplored. In this work, we make the first attempt at investigating the sensitivity of OPE methods to marginal adversarial perturbations to the data. We design a generic data poisoning attack framework leveraging influence functions from robust statistics to carefully construct perturbations that maximize error in the policy value estimates. We carry out extensive experimentation with multiple healthcare and control datasets. Our results demonstrate that many existing OPE methods are highly prone to generating value estimates with large errors when subject to data poisoning attacks, even for small adversarial perturbations. These findings question the reliability of policy values derived using OPE methods and motivate the need for developing OPE methods that are statistically robust to train-time data poisoning attacks.
BibTeX
@misc{lobo2024data,
  author = {Elita A. Lobo and Harvineet Singh and Marek Petrik and Cynthia Rudin and Himabindu Lakkaraju},
  title = {{Data Poisoning Attacks on Off-Policy Policy Evaluation Methods}},
  year = {2024},
  eprint = {2404.04714},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.04714},
}
Figure