Abstract
Reinforcement Learning (RL) enables agents to learn how to perform various tasks from scratch. In domains like autonomous driving, recommendation systems, and more, optimal RL policies learned could cause a privacy breach if the policies memorize any part of the private reward. We study the set of existing differentially-private RL policies derived from various RL algorithms such as Value Iteration, Deep-Q Networks, and Vanilla Proximal Policy Optimization. We propose a new Privacy-Aware Inverse RL analysis framework (PRIL) that involves performing reward reconstruction as an adversarial attack on private policies that the agents may deploy. For this, we introduce the reward reconstruction attack, wherein we seek to reconstruct the original reward from a privacy-preserving policy using the Inverse RL algorithm. An adversary must do poorly at reconstructing the original reward function if the agent uses a tightly private policy. Using this framework, we empirically test the effectiveness of the privacy guarantee offered by the private algorithms on instances of the FrozenLake domain of varying complexities. Based on the analysis performed, we infer a gap between the current standard of privacy offered and the standard of privacy needed to protect reward functions in RL. We do so by quantifying the extent to which each private policy protects the reward function by measuring distances between the original and reconstructed rewards.
Introduction
In Reinforcement Learning (RL), an agent learns to optimize its behavior by maximizing a reward function. In sensitive applications like personalized medicine or autonomous driving, these rewards often encode private user data. To mitigate privacy risks, researchers employ Differential Privacy (DP) to add noise during training, ensuring the resulting policy does not memorize specific data points. However, a critical question remains: does a mathematically private policy actually protect the underlying reward function? If an observer can infer high-reward zones by watching a "private" agent, the privacy guarantee is insufficient. This paper introduces the PRIL framework to measure this "privacy gap" by treating reward reconstruction as an adversarial attack.
Methods: The PRIL Framework
The authors propose the Privacy-Aware Inverse RL (PRIL) framework to audit the privacy of RL policies. The core concept is to use Inverse Reinforcement Learning (IRL)—traditionally used to learn goals from expert demonstrations—as an adversarial tool to reconstruct the original reward function from a private policy.
The workflow consists of three stages:
- Training: An agent learns a policy using private RL algorithms (e.g., Value Iteration, DQN, or PPO) with DP noise.
- Attack: An adversary applies an IRL algorithm (specifically Linear Programming IRL) to the private policy to reconstruct the reward function.
- Audit: The framework quantifies the attack's success by measuring the distance ($L_1, L_2, L_\infty$ norms and sign change counts) between the original and reconstructed rewards.
Experimental Setup
The framework was evaluated on 24 custom environments within the FrozenLake domain (OpenAI Gym). These environments featured grid sizes of 5x5 and 10x10 with diverse layouts, including safe paths, frozen tiles, penalties (holes), and high-reward goals. This variety ensured that the analysis was robust across different maze configurations.
Key Results and Insights
The study reveals a significant "privacy gap" in modern RL. The most striking finding is that the privacy budget ($\epsilon$), which controls the noise level, has minimal impact on an adversary's ability to reconstruct the reward. Even under "tight" privacy constraints (low $\epsilon$), reconstruction error remained relatively constant.
Key observations include:
- Reconstruction Success: Adversaries frequently reconstructed rewards that were visually and mathematically similar to the originals, despite high levels of DP noise in the policy.
- Policy Privacy $\neq$ Reward Privacy: Mathematical guarantees for policy privacy do not inherently protect the reward function. Because an agent must perform well to be useful, its actions inevitably leak information about its underlying objectives.
- Algorithm Variance: DQN generally provided better reward privacy than PPO or VI. This is likely because Deep RL learns generalized features, making it harder for IRL attacks to pinpoint exact reward values for specific states compared to tabular methods.
Discussion and Future Work
The results indicate that current DP-RL methods are largely ineffective against reward-reconstruction attacks. While increasing privacy (lower $\epsilon$) degrades agent performance, it does not provide a proportional increase in reward protection.
Future research should focus on:
- Scaling: Extending analysis to continuous state spaces common in robotics.
- Advanced Attacks: Testing against more sophisticated, non-linear IRL methods like Deep IRL.
- New Mechanisms: Developing DP mechanisms specifically designed to protect the reward function itself, rather than just the policy.
Summary updated