How Private Is Your RL Policy? An Inverse RL Based Analysis Framework

Kritika Prakash, Fiza Husain, Praveen Paruchuri, Sujit Gujar

Abstract

Reinforcement Learning (RL) enables agents to learn how to perform various tasks from scratch. In domains like autonomous driving and recommendation systems, optimal RL policies could cause a privacy breach if they memorize parts of the private reward. We study existing differentially-private RL policies derived from algorithms like Value Iteration, Deep Q-Networks, and Proximal Policy Optimization. We propose a new Privacy-Aware Inverse RL (PRIL) analysis framework that performs reward reconstruction as an adversarial attack on private policies. Using this framework, we empirically test the effectiveness of privacy guarantees on multiple FrozenLake environments. Our analysis reveals a significant gap between current privacy standards and the protection needed for reward functions in RL.

Introduction

In many real-world RL applications, the reward function is constructed from sensitive user data. For instance, a driving agent's rewards might reflect interactions with specific pedestrians, or a recommendation bot might learn from a user's private habits. While Differential Privacy (DP) aims to ensure policies do not leak specific training samples, it is less clear if these protections extend to the underlying reward function. If an adversary can observe a "private" policy and use Inverse Reinforcement Learning (IRL) to reverse-engineer the original reward, the user's privacy is compromised.

The PRIL Framework

The authors introduce PRIL, a framework that treats reward reconstruction as a specialized "data-reconstruction attack." The process involves three main steps:

  1. Training: An agent learns a policy ($P''$) using a private RL algorithm.
  2. Attacking: An adversary uses an IRL algorithm to reconstruct the reward function ($R''$) based solely on the observed private policy.
  3. Measuring: The framework calculates the distance (similarity) between the original reward ($R$) and the reconstructed reward ($R''$).
The PRIL framework uses Inverse RL as an adversarial attack to evaluate reward privacy.

If the privacy mechanism is effective, the reconstructed reward should differ significantly from the original. A low distance indicates a privacy failure.

Experimental Setup

The researchers evaluated PRIL on 24 custom environments within the FrozenLake domain, using grid sizes of 5x5 and 10x10. They tested three classes of RL algorithms:

  • Value Iteration (VI): Utilizing private Bellman updates.
  • Deep Q-Network (DQN): Using DP optimizers (DP-SGD, DP-Adam) and functional noise.
  • Proximal Policy Optimization (PPO): Applying DP to the actor network.

Results and the "Privacy Gap"

The most striking finding is the existence of a privacy gap: the reconstruction error is often independent of the privacy budget ($\epsilon$). Even with high noise (low $\epsilon$), attackers could often reconstruct a reward function very close to the original.

Qualitative comparison showing that reconstructed rewards (bottom) often closely resemble original rewards (top) despite privacy measures.

Key insights include:

  • Policy Privacy $\neq$ Reward Privacy: A policy that is mathematically "private" by DP standards can still leak the underlying intent (the reward).
  • Reconstruction Accuracy: In many cases, the adversary could clearly detect obstacles and rewarding paths in the reconstructed heatmaps, even when the policy was trained with high noise.
  • Algorithm Sensitivity: DQN generally offered better reward protection than PPO or Value Iteration. Specifically, the "DP-Shoe" approach (using tanh activations and standard SGD) performed better than ReLU and Adam optimizers.

Conclusion and Future Work

The PRIL framework demonstrates that current DP-RL methods are "privacy agnostic" regarding the reward domain—they do not necessarily make the reward harder to find as privacy settings increase. This study highlights the need for new RL algorithms that explicitly protect the reward function. Future work includes scaling these attacks to continuous state spaces and exploring "black-box" attacks where environment dynamics are unknown to the adversary.

Summary updated

Abstract

Reinforcement Learning (RL) enables agents to learn how to perform various tasks from scratch. In domains like autonomous driving, recommendation systems, and more, optimal RL policies learned could cause a privacy breach if the policies memorize any part of the private reward. We study the set of existing differentially-private RL policies derived from various RL algorithms such as Value Iteration, Deep Q Networks, and Vanilla Proximal Policy Optimization. We propose a new Privacy-Aware Inverse RL (PRIL) analysis framework, that performs reward reconstruction as an adversarial attack on private policies that the agents may deploy. For this, we introduce the reward reconstruction attack, wherein we seek to reconstruct the original reward from a privacy-preserving policy using an Inverse RL algorithm. An adversary must do poorly at reconstructing the original reward function if the agent uses a tightly private policy. Using this framework, we empirically test the effectiveness of the privacy guarantee offered by the private algorithms on multiple instances of the FrozenLake domain of varying complexities. Based on the analysis performed, we infer a gap between the current standard of privacy offered and the standard of privacy needed to protect reward functions in RL. We do so by quantifying the extent to which each private policy protects the reward function by measuring distances between the original and reconstructed rewards.
BibTeX
@inproceedings{prakash2022how,
  author = {Kritika Prakash and Fiza Husain and Praveen Paruchuri and Sujit Gujar},
  title = {{How Private Is Your RL Policy? An Inverse RL Based Analysis Framework}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=kBTLrW1Onc},
}
Figure