Abstract
Reinforcement Learning (RL) enables agents to learn how to perform various tasks from scratch. In domains like autonomous driving and recommendation systems, optimal RL policies could cause a privacy breach if they memorize parts of the private reward. We study existing differentially-private RL policies derived from algorithms like Value Iteration, Deep Q-Networks, and Proximal Policy Optimization. We propose a new Privacy-Aware Inverse RL (PRIL) analysis framework that performs reward reconstruction as an adversarial attack on private policies. Using this framework, we empirically test the effectiveness of privacy guarantees on multiple FrozenLake environments. Our analysis reveals a significant gap between current privacy standards and the protection needed for reward functions in RL.
Introduction
In many real-world RL applications, the reward function is constructed from sensitive user data. For instance, a driving agent's rewards might reflect interactions with specific pedestrians, or a recommendation bot might learn from a user's private habits. While Differential Privacy (DP) aims to ensure policies do not leak specific training samples, it is less clear if these protections extend to the underlying reward function. If an adversary can observe a "private" policy and use Inverse Reinforcement Learning (IRL) to reverse-engineer the original reward, the user's privacy is compromised.
The PRIL Framework
The authors introduce PRIL, a framework that treats reward reconstruction as a specialized "data-reconstruction attack." The process involves three main steps:
- Training: An agent learns a policy ($P''$) using a private RL algorithm.
- Attacking: An adversary uses an IRL algorithm to reconstruct the reward function ($R''$) based solely on the observed private policy.
- Measuring: The framework calculates the distance (similarity) between the original reward ($R$) and the reconstructed reward ($R''$).
If the privacy mechanism is effective, the reconstructed reward should differ significantly from the original. A low distance indicates a privacy failure.
Experimental Setup
The researchers evaluated PRIL on 24 custom environments within the FrozenLake domain, using grid sizes of 5x5 and 10x10. They tested three classes of RL algorithms:
- Value Iteration (VI): Utilizing private Bellman updates.
- Deep Q-Network (DQN): Using DP optimizers (DP-SGD, DP-Adam) and functional noise.
- Proximal Policy Optimization (PPO): Applying DP to the actor network.
Results and the "Privacy Gap"
The most striking finding is the existence of a privacy gap: the reconstruction error is often independent of the privacy budget ($\epsilon$). Even with high noise (low $\epsilon$), attackers could often reconstruct a reward function very close to the original.
Key insights include:
- Policy Privacy $\neq$ Reward Privacy: A policy that is mathematically "private" by DP standards can still leak the underlying intent (the reward).
- Reconstruction Accuracy: In many cases, the adversary could clearly detect obstacles and rewarding paths in the reconstructed heatmaps, even when the policy was trained with high noise.
- Algorithm Sensitivity: DQN generally offered better reward protection than PPO or Value Iteration. Specifically, the "DP-Shoe" approach (using
tanhactivations and standard SGD) performed better than ReLU and Adam optimizers.
Conclusion and Future Work
The PRIL framework demonstrates that current DP-RL methods are "privacy agnostic" regarding the reward domain—they do not necessarily make the reward harder to find as privacy settings increase. This study highlights the need for new RL algorithms that explicitly protect the reward function. Future work includes scaling these attacks to continuous state spaces and exploring "black-box" attacks where environment dynamics are unknown to the adversary.
Summary updated