Preventing Imitation Learning with Adversarial Policy Ensembles

Albert Zhan, Stas Tiomkin, Pieter AbbeelarXiv 2020 · arXiv

TL;DR

This paper introduces Adversarial Policy Ensembles (APE), a novel reinforcement learning framework designed to protect proprietary robotic or software policies from being stolen via imitation learning. By training an ensemble of near-optimal policies that rely on a hidden context variable, the system maintains high performance for the owner while ensuring that any cloned policy is completely crippled.

What Problem Does This Paper Address?

Imitation learning—specifically behavioral cloning (BC)—has become incredibly powerful at replicating complex behaviors by simply observing an expert. While this is great for training robots efficiently, it poses a massive intellectual property and privacy risk. If you deploy a proprietary navigation or control policy on a commercial robot, an external observer could record the robot's state-action pairs and train their own clone without your consent.

Prior work in machine learning privacy has focused on protecting datasets or classification models, but protecting sequential decision-making policies from being cloned remained an open challenge. This paper tackles a fundamental question: Can we design an expert policy that performs a task well, but is intentionally structured to be impossible to clone accurately?

Methods

The core idea relies on an information asymmetry between the policy owner and the malicious observer, formalized as a Policy Ensemble (PE).

The owner trains an ensemble of expert policies, denoted as $\pi_c(a|s,c)$, where $c$ is a latent context variable (or "style") randomly chosen at the start of each trajectory. Crucially, while the owner knows which context $c$ is active, an external observer trying to clone the policy only sees the raw states and actions. Because the observer lacks access to $c$, their behavioral cloning algorithm is forced to average out (marginalize) the context, resulting in a fictitious blended policy:

$$\pi_o(a | s) := \sum_{i} p(c^{(i)} | s)\pi_{c^{(i)}}(a | s)$$

Figure 1: Confidentiality scheme: Left During training, optimize a Policy Ensemble by estimating gradients using both the policies in the ensemble and the fictitious observer policy. Right When collecting a dataset for cloning, the context variable is marginalized out. Thus cloning the Policy Ensemble can result in a useless policy

To exploit this, the authors modify the standard Reinforcement Learning objective into a constrained optimization problem using a Lagrangian formulation:

$$J(\theta) = \mathbb{E}_{\tau \sim \rho_{\pi_c}}[r(\tau)] - \beta \mathbb{E}_{\tau \sim \rho_{\pi_o}}[r(\tau)]$$

Here, $\beta$ is a weighting factor. The objective maximizes the reward of the true policy ensemble ($\pi_c$) while actively minimizing the expected reward of the observer's marginalized policy ($\pi_o$). The authors introduce PG-APE (Algorithm 1), a modified Policy Gradient algorithm that simultaneously samples trajectories from both the true ensemble and the virtual observer policy to update the policy parameters.

Datasets

Because this is a foundational paper introducing a new paradigm, the authors evaluate their method on a controlled 10×10 Grid-World navigation task rather than a standard static dataset.

In this environment, an agent spawns randomly and must reach a goal state in the top-left corner as fast as possible, receiving a reward of -1 for every step taken. This discrete environment was deliberately chosen because it allows for precise, analytical calculations of expected returns, eliminating confounding factors like approximation errors or distributional drift.

Key Insights and Ideas

  • Intentional Sub-optimality as a Shield: To confuse the cloner, the APE experts learn to make choices that look contradictory when averaged together. For instance, near the goal, one expert might choose to take an "intelligent detour" while another chooses to "Stay" or move in an alternate path.
  • The Power of Contradiction: Individually, each expert policy is "good enough" to reach the goal efficiently. However, when the observer blends them together into $\pi_o$, these conflicting actions cancel each other out or create loops, trapping the cloned agent in a state of perpetual confusion.
  • Passive Protection: Unlike active adversarial defenses that inject noise into a system at runtime, APE provides passive protection. The deployed policy runs naturally; its inherent structure is what defeats the cloner.

Key Results

The experimental results demonstrate a massive performance gap between the true ensemble and the stolen clone:

  • Ensemble Performance: The APE ensemble achieved a solid average return of -16.24, which is very close to a standard near-optimal policy (-16.74).
  • Clone Degradation: The policy obtained by cloning the APE ensemble was severely crippled, scoring an average return of -44.27. This represents more than a 5× performance degradation compared to the absolute optimal policy (-9).
Figure 3: Visualization of the cloned APE. The policy obtained from cloning the APE trained has av- erage expected reward of −45.18, while the optimal policy has an average expected reward of −9, which is over a 5× increase.

As shown in the baseline comparison table below, standard training methods ("Near-Optimal PE") result in clones that perform just as well as the original policy. Only APE successfully creates a massive "Returns Difference" (-28.03), proving that anti-cloning protection is a non-trivial property that must be explicitly optimized.

Quantitative comparison table against baselines

Limitations and Future Work

While a vital first step, the authors note a few key limitations:

  1. The Sequential Observer Loophole: The current defense assumes the observer maps single states to actions. If a thief uses a sequential model (like an RNN) to look at a history of states, they might successfully infer the hidden context variable $c$ over time. Preliminary tests showed that if the observer's representation is strong enough, the defense degrades.
  2. Discrete State Spaces: The method was only validated on a discrete grid-world. Extending this framework to continuous control domains (like MuJoCo or real-world robotics) requires scaling the context distribution estimation, which remains future work.

Summary updated

Abstract

Imitation learning can reproduce policies by observing experts, which poses a problem regarding policy privacy. Policies, such as human, or policies on deployed robots, can all be cloned without consent from the owners. How can we protect against external observers cloning our proprietary policies? To answer this question we introduce a new reinforcement learning framework, where we train an ensemble of near-optimal policies, whose demonstrations are guaranteed to be useless for an external observer. We formulate this idea by a constrained optimization problem, where the objective is to improve proprietary policies, and at the same time deteriorate the virtual policy of an eventual external observer. We design a tractable algorithm to solve this new optimization problem by modifying the standard policy gradient algorithm. Our formulation can be interpreted in lenses of confidentiality and adversarial behaviour, which enables a broader perspective of this work. We demonstrate the existence of "non-clonable" ensembles, providing a solution to the above optimization problem, which is calculated by our modified policy gradient algorithm. To our knowledge, this is the first work regarding the protection of policies in Reinforcement Learning.
BibTeX
@misc{zhan2020preventing,
  author = {Albert Zhan and Stas Tiomkin and Pieter Abbeel},
  title = {{Preventing Imitation Learning with Adversarial Policy Ensembles}},
  year = {2020},
  eprint = {2002.01059},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2002.01059},
}
Figure