How Private Is Your RL Policy? An Inverse RL Based Analysis Framework

Kritika Prakash, Fiza Husain, Praveen Paruchuri, Sujit Gujar

Abstract

Reinforcement Learning (RL) enables agents to learn how to perform various tasks from scratch. In domains like autonomous driving, recommendation systems, and more, optimal RL policies learned could cause a privacy breach if the policies memorize any part of the private reward. We study the set of existing differentially-private RL policies derived from various RL algorithms such as Value Iteration, Deep-Q Networks, and Vanilla Proximal Policy Optimization. We propose a new Privacy-Aware Inverse RL analysis framework (PRIL) that involves performing reward reconstruction as an adversarial attack on private policies that the agents may deploy. For this, we introduce the reward reconstruction attack, wherein we seek to reconstruct the original reward from a privacy-preserving policy using the Inverse RL algorithm. An adversary must do poorly at reconstructing the original reward function if the agent uses a tightly private policy. Using this framework, we empirically test the effectiveness of the privacy guarantee offered by the private algorithms on instances of the FrozenLake domain of varying complexities. Based on the analysis performed, we infer a gap between the current standard of privacy offered and the standard of privacy needed to protect reward functions in RL. We do so by quantifying the extent to which each private policy protects the reward function by measuring distances between the original and reconstructed rewards.

Introduction

In Reinforcement Learning (RL), an agent learns to optimize its behavior by maximizing a reward function. In sensitive applications like personalized medicine or autonomous driving, these rewards often encode private user data. To mitigate privacy risks, researchers employ Differential Privacy (DP) to add noise during training, ensuring the resulting policy does not memorize specific data points. However, a critical question remains: does a mathematically private policy actually protect the underlying reward function? If an observer can infer high-reward zones by watching a "private" agent, the privacy guarantee is insufficient. This paper introduces the PRIL framework to measure this "privacy gap" by treating reward reconstruction as an adversarial attack.

Methods: The PRIL Framework

The authors propose the Privacy-Aware Inverse RL (PRIL) framework to audit the privacy of RL policies. The core concept is to use Inverse Reinforcement Learning (IRL)—traditionally used to learn goals from expert demonstrations—as an adversarial tool to reconstruct the original reward function from a private policy.

The PRIL framework workflow: Original rewards are processed through private RL algorithms to produce private policies, which are then subjected to Inverse RL to generate reconstructed rewards for comparison.

The workflow consists of three stages:

  1. Training: An agent learns a policy using private RL algorithms (e.g., Value Iteration, DQN, or PPO) with DP noise.
  2. Attack: An adversary applies an IRL algorithm (specifically Linear Programming IRL) to the private policy to reconstruct the reward function.
  3. Audit: The framework quantifies the attack's success by measuring the distance ($L_1, L_2, L_\infty$ norms and sign change counts) between the original and reconstructed rewards.

Experimental Setup

The framework was evaluated on 24 custom environments within the FrozenLake domain (OpenAI Gym). These environments featured grid sizes of 5x5 and 10x10 with diverse layouts, including safe paths, frozen tiles, penalties (holes), and high-reward goals. This variety ensured that the analysis was robust across different maze configurations.

Key Results and Insights

The study reveals a significant "privacy gap" in modern RL. The most striking finding is that the privacy budget ($\epsilon$), which controls the noise level, has minimal impact on an adversary's ability to reconstruct the reward. Even under "tight" privacy constraints (low $\epsilon$), reconstruction error remained relatively constant.

Visual comparison of original rewards (top row) versus rewards recovered from private policies (bottom row) across different environment configurations.

Key observations include:

  • Reconstruction Success: Adversaries frequently reconstructed rewards that were visually and mathematically similar to the originals, despite high levels of DP noise in the policy.
  • Policy Privacy $\neq$ Reward Privacy: Mathematical guarantees for policy privacy do not inherently protect the reward function. Because an agent must perform well to be useful, its actions inevitably leak information about its underlying objectives.
  • Algorithm Variance: DQN generally provided better reward privacy than PPO or VI. This is likely because Deep RL learns generalized features, making it harder for IRL attacks to pinpoint exact reward values for specific states compared to tabular methods.

Discussion and Future Work

The results indicate that current DP-RL methods are largely ineffective against reward-reconstruction attacks. While increasing privacy (lower $\epsilon$) degrades agent performance, it does not provide a proportional increase in reward protection.

Future research should focus on:

  • Scaling: Extending analysis to continuous state spaces common in robotics.
  • Advanced Attacks: Testing against more sophisticated, non-linear IRL methods like Deep IRL.
  • New Mechanisms: Developing DP mechanisms specifically designed to protect the reward function itself, rather than just the policy.

Summary updated

Abstract

Reinforcement Learning (RL) enables agents to learn how to perform various tasks from scratch. In domains like autonomous driving, recommendation systems, and more, optimal RL policies learned could cause a privacy breach if the policies memorize any part of the private reward. We study the set of existing differentially-private RL policies derived from various RL algorithms such as Value Iteration, Deep-Q Networks, and Vanilla Proximal Policy Optimization. We propose a new Privacy-Aware Inverse RL analysis framework (PRIL) that involves performing reward reconstruction as an adversarial attack on private policies that the agents may deploy. For this, we introduce the reward reconstruction attack, wherein we seek to reconstruct the original reward from a privacy-preserving policy using the Inverse RL algorithm. An adversary must do poorly at reconstructing the original reward function if the agent uses a tightly private policy. Using this framework, we empirically test the effectiveness of the privacy guarantee offered by the private algorithms on instances of the FrozenLake domain of varying complexities. Based on the analysis performed, we infer a gap between the current standard of privacy offered and the standard of privacy needed to protect reward functions in RL. We do so by quantifying the extent to which each private policy protects the reward function by measuring distances between the original and reconstructed rewards.
BibTeX
@inproceedings{prakash2022how,
  author = {Kritika Prakash and Fiza Husain and Praveen Paruchuri and Sujit Gujar},
  title = {{How Private Is Your RL Policy? An Inverse RL Based Analysis Framework}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=kBTLrW1Onc},
}
Figure