Model Extraction Attacks Against Reinforcement Learning Based Controllers

Momina Sajid, Yanning Shen, Yasser Shoukry

Introduction

As Cyber-Physical Systems (CPS)—ranging from autonomous robotics to critical power grid infrastructure—increasingly integrate Deep Neural Networks (DNNs) as their primary control logic, the security of these "brains" becomes paramount. While Reinforcement Learning (RL) has proven highly effective at mastering complex control tasks, it introduces a significant security vulnerability: model extraction. This paper, authored by researchers at the University of California, Irvine, investigates how an adversary can steal the parameters of a black-box RL controller, potentially leading to intellectual property theft or the facilitation of devastating physical attacks.

The Challenge of RL Model Extraction

In a standard model extraction attack, an adversary attempts to "clone" a proprietary model by observing its input-output behavior. While well-documented in supervised learning contexts (such as image classification), extracting RL-based controllers presents unique difficulties:

  1. Data Scarcity and Observability: Unlike a public API, a CPS controller often operates in a restricted environment. An attacker may only observe a narrow slice of the system's state space and cannot typically provide arbitrary inputs to the model.
  2. Non-Convexity and Local Optima: RL training is a non-convex optimization process. Multiple distinct sets of neural network weights (parameters $\theta$) can result in similar performance or represent different local maxima for the same reward function. This makes it mathematically difficult to pinpoint the exact controller used by the target system through observation alone.

Methodology: A Two-Phase Attack Framework

To overcome these hurdles, the authors propose a novel algorithm that bifurcates the attack into an offline preparation stage and an online refinement stage.

Phase 1: Offline Candidate Generation

The attacker begins by leveraging "side-channel" information, such as a noisy approximation of the reward function and the known physical dynamics of the system.

  • Gradient Landscape Approximation: Using multi-kernel regression, the attacker maps the potential optimization landscape of the RL problem.
  • Formal Methods Integration: The researchers employ Satisfiability Modulo Theory (SMT) solvers—tools traditionally used in formal software verification—to identify all sets of parameters ($\theta$) that function as "critical points" (local or global maxima) within that landscape. This phase generates a broad pool of thousands of potential "candidate" policies that the target might be using.

Phase 2: Online Shortlisting

Once the target system is operational, the attacker transition to active observation. By monitoring real-time state-action pairs $(s, a)$, the attacker filters the candidate pool:

  • Pruning: For every new observation, each candidate policy is tested. If a candidate's predicted action deviates from the observed action by more than a predefined threshold ($\psi$), it is discarded.
  • Convergence: This process rapidly narrows the field from thousands of possibilities to a handful of highly accurate models that are functionally indistinguishable from the original.

Experimental Evaluation

The attack was validated using two benchmarks from the OpenAI Gym, simulated via the Box2D physics engine:

  • Inverted Pendulum: A classic linear control problem involving balancing a pole.
  • Mountain Car: A non-linear environment requiring the car to build momentum. This environment is particularly challenging due to its numerous local optima.

Key Results

The results demonstrate that the attack is both data-efficient and highly accurate:

Environment Initial Candidates Final Candidates (Avg) Samples Needed Max Error
Inverted Pendulum 2,047 4.24 150 0.037
Mountain Car 9,136 6.00 150 0.463

In both cases, the online phase eliminated over 99% of incorrect candidates within just 150 observations. The remaining models exhibited errors well below the attacker's target thresholds, effectively granting the adversary a functional clone of the proprietary controller.

Discussion and Insights

The research highlights several critical takeaways for CPS security:

  • The Vulnerability of Reward Functions: The attack proves that an adversary does not need the exact reward function used during training; a noisy approximation is sufficient to generate a viable candidate pool.
  • Efficiency of Formal Methods: The use of SMT solvers provides a powerful bridge between control theory and logic, allowing attackers to solve feasibility problems that would be computationally prohibitive using standard search methods.
  • Minimal Data Requirements: Because the offline phase performs the heavy lifting of mapping the optimization landscape, the online phase requires very little data to succeed, making the attack difficult to detect through traffic analysis.

Limitations and Future Directions

The primary constraint of this method is the requirement for knowledge regarding the reward structure and system dynamics. While these are often standardized in specific industries (e.g., power grid frequency control), a completely opaque system would resist the offline phase. Additionally, the computational complexity of SMT solvers may limit the attack's applicability to very large neural networks with millions of parameters.

Future research may explore defensive strategies, such as "policy obfuscation," where small amounts of strategic noise are added to control actions to prevent attackers from accurately pruning their candidate sets.

Summary updated

BibTeX
@inproceedings{sajid2023model,
  author = {Momina Sajid and Yanning Shen and Yasser Shoukry},
  title = {{Model Extraction Attacks Against Reinforcement Learning Based Controllers}},
  booktitle = {CDC},
  year = {2023},
  doi = {10.1109/cdc49753.2023.10383661},
}
Figure