Reinforcement Learning on Encrypted Data

Alberto Jesu, Victor-Alexandru Darvariu, Alessandro Staffolani, Rebecca Montanari, Mirco Musolesi

TL;DR

This paper explores the feasibility of training Reinforcement Learning (RL) agents on encrypted data, a crucial step for privacy in sensitive domains like industrial control or healthcare. They propose a modified MDP framework and evaluate a DQN agent's performance across various encryption schemes and environments, finding that while simple encryption can work for small state spaces, complex environments pose significant challenges.

What Problem Does This Paper Address?

Reinforcement Learning (RL) is increasingly applied to real-world problems, but many of these domains involve sensitive data. While differential privacy has been a focus for protecting data during training, this paper addresses a more stringent scenario: when the input data itself must remain encrypted, even from the learning agent. This is particularly relevant for applications like industrial control systems or remote monitoring where data confidentiality is paramount, and sharing raw data is not an option. The challenge lies in enabling RL algorithms to learn effective policies when their sensory input is encrypted, potentially obscuring crucial information.

Methods

The authors extend the standard Markov Decision Process (MDP) framework to accommodate encrypted states. This involves adding a state processing step and an encryption primitive before the state is fed to the agent. They specifically adapt the Deep Q-Network (DQN) algorithm, a popular RL method known for its sample efficiency. The core modification is an additional layer that preprocesses and encrypts the state, ensuring the agent only receives encrypted inputs. The encrypted states are stored in the replay memory and used for network updates.

For encryption, they experiment with:

  • Block Ciphers: AES in ECB and CBC modes.
  • Homomorphic Encryption (HE): Specifically, the CKKS scheme, which allows computations on encrypted data.
  • Baselines: A "no-op" primitive (plaintext) and a deterministic shuffling of image pixels.

They also investigate the impact of padding schemes (custom and PKCS#7) and key lengths for block ciphers.

Datasets

The experiments were conducted using environments from OpenAI Gym:

  • MiniGrid: A discrete navigation task with varying grid sizes (5x5, 6x6, 8x8, 16x16). Observations are RGB images.
  • LunarLander: A continuous control task. Observations are 8-dimensional arrays.

No new datasets were introduced; existing benchmark environments were adapted to incorporate the encryption pipeline.

Key Insights and Ideas

  • Homomorphic Encryption is Currently Impractical for RL: The significant computational overhead of HE schemes makes them infeasible for current RL algorithms on commodity hardware, especially given the sample complexity of RL.
  • Deterministic Encryption Can Be as Good as Plaintext: For small, discrete state spaces (like MiniGrid), deterministic encryption methods (e.g., AES in ECB mode) can achieve comparable performance to training on plaintext data.
  • Non-Deterministic Encryption Can Be Surprisingly Robust (in specific cases): In MiniGrid, AES with CBC mode, despite being non-deterministic and not reaching optimal solutions, still outperformed random actions, suggesting a degree of robustness.
  • Complexity is the Enemy: As state spaces grow larger and control signals become continuous (as in LunarLander), the ability of RL agents to learn from encrypted data degrades significantly, with none of the tested encryption schemes succeeding in these complex scenarios.
  • Padding and Key Length Have Limited Impact on Small State Spaces: For the MiniGrid environment, variations in padding schemes and key lengths for block ciphers did not lead to statistically significant differences in performance, likely due to the limited state space.

Key Results

  • MiniGrid:
    • AES in ECB mode performed on par with plaintext and deterministic shuffle across various grid sizes.
  • AES in CBC mode showed some learning capability but did not reach optimal solutions, outperforming random baselines.
  • Encryption overhead for block ciphers was negligible compared to HE.
  • LunarLander:
    • The agent trained on encrypted data (even with state discretization) showed no learning and progressively worse performance compared to the plaintext case, which itself diverged. This highlights the difficulty of learning in continuous action spaces with encrypted inputs.
  • Encryption Time: HE was orders of magnitude slower than block ciphers and shuffling.

Limitations and Future Work

Limitations: * Scalability: The primary limitation is the significant performance degradation in complex environments with large state spaces and continuous actions. The proposed methods did not succeed in these scenarios. * HE Feasibility: The current computational cost of HE makes it impractical for RL on standard hardware.

Future Work: * Investigating More Sophisticated HE Schemes: Exploring newer or optimized HE schemes that might reduce computational overhead. * Developing Encryption-Aware RL Algorithms: Designing RL algorithms that are inherently more robust to noise or information loss introduced by encryption. * Exploring Hybrid Approaches: Combining different privacy techniques or encryption methods to find a better balance between privacy and performance. * Addressing Complex Environments: Further research is needed to enable RL on encrypted data in more challenging, real-world scenarios.

Summary updated

Abstract

The growing number of applications of Reinforcement Learning (RL) in real-world domains has led to the development of privacy-preserving techniques due to the inherently sensitive nature of data. Most existing works focus on differential privacy, in which information is revealed in the clear to an agent whose learned model should be robust against information leakage to malicious third parties. Motivated by use cases in which only encrypted data might be shared, such as information from sensitive sites, in this work we consider scenarios in which the inputs themselves are sensitive and cannot be revealed. We develop a simple extension to the MDP framework which provides for the encryption of states. We present a preliminary, experimental study of how a DQN agent trained on encrypted states performs in environments with discrete and continuous state spaces. Our results highlight that the agent is still capable of learning in small state spaces even in presence of non-deterministic encryption, but performance collapses in more complex environments.
BibTeX
@misc{jesu2021reinforcement,
  author = {Alberto Jesu and Victor-Alexandru Darvariu and Alessandro Staffolani and Rebecca Montanari and Mirco Musolesi},
  title = {{Reinforcement Learning on Encrypted Data}},
  year = {2021},
  eprint = {2109.08236},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.08236},
}
Figure