Projections for Approximate Policy Iteration Algorithms

Riad Akrour, Joni Pajarinen, Jan Peters, Gerhard NeumannICML 2019 · OpenReview

Introduction

In Reinforcement Learning (RL), "moving too fast" can be fatal. When an agent updates its policy, if the new policy is drastically different from the old one, the data it collects might change so much that the learning process collapses. To prevent this, researchers use constraints:

  1. KL Divergence Constraint: Ensures the new policy stays "close" to the old one, maintaining stability.
  2. Entropy Constraint: Ensures the policy doesn't become too deterministic too quickly, preserving exploration.

Traditionally, solving these constrained problems requires the method of Lagrange multipliers (which often needs sample-based approximations and can be unstable) or quadratic approximations (like in Trust Region Policy Optimization, or TRPO). These methods can be computationally heavy or fail when sample sizes are small.

This paper introduces a clever way to handle these "safety rails" using differentiable projections. Instead of using complex mathematical workarounds to keep policy updates stable, the authors derive mapping functions that transform constrained optimization problems into unconstrained ones, making algorithms like TRPO and PPO more robust, sample-efficient, and easier to tune.

Differentiable Projections

The core innovation is the derivation of a projection function $g$.

Instead of solving $\max f(\pi)$ subject to constraints, the proposed method solves $\max f(g(\theta))$. The function $g$ maps any arbitrary parameter $\theta$ into a subspace that guarantees the constraints are met.

  • Differentiability: Because $g$ is differentiable, standard backpropagation and gradient descent can be used on the combined function $f \circ g$.
  • Direct Policy Search (DPS): The authors derive projections for Gaussian search distributions, ensuring the mean and covariance stay within KL and entropy bounds.
  • Approximate Policy Iteration (API): This is extended to state-conditioned policies (like neural networks). The authors specifically focus on a "linear-in-features" setup for the final layer of a policy network to make the projection computationally practical.

This approach fundamentally differs from standard "Projected Gradient Descent," which allows a step to go outside the boundary and then pushes it back. Instead, this method reparameterizes the space so the optimizer is always moving along the "manifold" of valid policies.

Experimental Setup

The authors evaluated their approach across several benchmarks to test scalability and exploration:

  • Synthetic Smooth Functions: Mixtures of 25 bivariate Gaussians to test Direct Policy Search.
  • Roboschool (v1): A suite of continuous control tasks including Hopper, Walker, HalfCheetah, and Ant.
  • BitFlip: A discrete action task specifically designed to require sustained exploration, where greedy behavior is sub-optimal.

Key Findings

By turning a constrained problem into an unconstrained one, the authors sidestep the need to solve the "dual problem" (finding the right Lagrange multipliers), which is often the most fragile part of RL implementations.

Optimization Quality and Stability Quantitative evaluations show a stark difference in how the proposed method navigates the optimization landscape compared to baselines. While projected gradient methods "bounce" around the constraint boundaries, the proposed projection method smoothly drives the optimizer toward the optimal point while strictly respecting the constraints.

Optimization trajectory comparing the proposed projection method against projected gradient descent.

Furthermore, the projection method demonstrates significantly lower regret and strictly adheres to KL constraints. In contrast, projected gradient baselines suffer from high variance and frequent constraint violations.

Box plots showing Regret and KL divergence, highlighting the stability of the proposed method.

Algorithm Performance Improvements * Robustness to Low Samples: In the DPS experiments, the projection method (termed "ProjectionPS") significantly outperformed existing methods like REPS and MORE when the number of samples per iteration was low, suggesting the projection provides a stronger "prior" for the update. * Better than Quadratic Approximations: In TRPO experiments, the projection method found policies closer to the actual KL constraint boundary and achieved higher objective values than TRPO’s standard Taylor-series approximation. * PAPI-TRPO and PAPI-PPO: The "Projected" versions of these famous algorithms showed clear improvements. In Roboschool tasks, PAPI-PPO often achieved higher average rewards in the initial 100 iterations compared to standard PPO. * BitFlip Success: On the BitFlip task, standard TRPO often converged early to sub-optimal policies. The PAPI variants, using the strict entropy projection, maintained exploration longer and eventually reached significantly higher final rewards.

Limitations and Future Work

While highly effective, the approach has limitations. The primary derivations rely on Gaussian assumptions; while a discrete version is provided, the most powerful results are for continuous action spaces. Additionally, for the neural network (API) case, the projection is applied only to the final linear layer. While it doesn't strictly constrain the updates of the deep hidden layers, the authors mitigate this by adjusting the learning rate.

Future work could extend this framework to other types of constraints—such as safety constraints or sparsity—and integrate it into a wider variety of RL architectures beyond TRPO and PPO. Ultimately, thinking of these projections as a "constraint layer" within a policy network offers a promising, simplified path forward for safe and stable RL.

Summary updated

Abstract

Approximate policy iteration is a class of reinforcement learning (RL) algorithms where the policy is encoded using a function approximator and which has been especially prominent in RL with contin...
BibTeX
@inproceedings{akrour2019projections,
  author = {Riad Akrour and Joni Pajarinen and Jan Peters and Gerhard Neumann},
  title = {{Projections for Approximate Policy Iteration Algorithms}},
  booktitle = {International Conference on Machine Learning},
  year = {2019},
  url = {https://openreview.net/forum?id=r1-gXhWdWB},
}
Figure