Abstract
Off-Policy Evaluation (OPE) is a cornerstone of practical reinforcement learning, enabling the estimation of a new policy's performance using historical data. However, the security of OPE methods against adversarial data manipulation remains under-explored. This paper introduces DOPE (Data poisoning attacks on Off-Policy Evaluation), a framework that demonstrates how subtle, targeted perturbations to a small fraction of logged data can lead to catastrophic errors in value estimation. Our findings reveal that corrupting as little as 3% of a dataset can result in errors exceeding 300%, particularly in high-stakes domains like healthcare.
Introduction
In many real-world applications of reinforcement learning, such as medical treatment planning or financial trading, direct experimentation is often unethical or prohibitively expensive. OPE methods bridge this gap by providing value estimates from static, historical datasets. These methods fundamentally rely on the assumption that the logged data is an accurate reflection of the environment. This research investigates the impact of violating this assumption through "train-time" data poisoning. We show that an attacker can introduce unnoticeable "poison" into the data to trick stakeholders into deploying suboptimal or dangerous policies.
The DOPE Framework
The authors propose the DOPE framework, which formulates data poisoning as a bi-level optimization problem. The attacker's goal is to maximize the error in the final policy value estimate while keeping perturbations small enough to remain stealthy and limiting the attack to a small fraction of the total data.
To overcome the NP-Hard nature of selecting optimal data subsets for corruption, DOPE utilizes influence functions from robust statistics. By calculating the influence of individual data points on the final OPE estimate, the attacker can efficiently identify and perturb the transitions that cause the most significant damage. The framework evaluates vulnerabilities across five major OPE categories: Direct Methods (BRM), Importance Sampling (IS) Methods (WIS, PDIS, CPDIS), and Hybrid Methods (WDR).
Experimental Setup
The DOPE framework was rigorously tested across five diverse environments: healthcare simulations (Cancer and HIV) and control tasks (Mountain Car, Cartpole, and a custom Continuous Gridworld). These domains represent a spectrum from low-dimensional control to complex medical scenarios where OPE is a critical tool for decision-making.
Vulnerability Analysis
A key insight from this work is the inherent brittleness of many OPE methods. Errors in value estimation often compound; a small perturbation in a future state's estimated value propagates backward, significantly affecting the initial state's value estimate.
Interestingly, the research found that WIS and CPDIS are notably more robust than their counterparts. This resilience is attributed to weight normalization, which limits the influence any single poisoned trajectory can exert on the overall average. Furthermore, the optimized nature of these attacks makes them difficult to detect using standard anomaly detection tools.
Key Results
The empirical results underscore the potency of targeted data poisoning. In the HIV domain, a mere 5% corruption of observed states led to a staggering 340% error in the value estimate. Similarly, in the Mountain Car environment, the same level of corruption resulted in a 100% error.
Methods such as BRM, PDIS, and WDR were consistently the most vulnerable. In extreme cases, such as the Cartpole environment, PDIS errors reached astronomical scales, rendering the estimates completely useless. DOPE consistently outperformed baseline attacks, including random noise and standard gradient-based methods, proving that influence-based poisoning is a sophisticated and effective threat.
Discussion and Future Work
While DOPE exposes critical vulnerabilities, the current framework relies on linear function approximation for calculating influence functions. Future work could extend the framework to fully end-to-end deep RL models. Additionally, the current attack assumes "white-box" access to the evaluation policy and OPE method; exploring "black-box" scenarios remains an open challenge. Ultimately, this work serves as a call to action for the RL community to develop OPE algorithms that are statistically robust to adversarial threats, ensuring safe deployment in high-stakes environments.
Summary updated