Evolving Constrained Reinforcement Learning Policy

Chengpeng Hu, Jiyuan Pei, Jialin Liu, Xin Yao

Introduction

Reinforcement Learning (RL) has demonstrated significant potential in solving complex control tasks. However, it often struggles with efficient exploration and "temporal credit assignment"—the challenge of attributing rewards to specific past actions. Evolutionary Reinforcement Learning (ERL) addresses these issues by maintaining a population of diverse actors. Yet, applying ERL to real-world scenarios introduces a new challenge: safety constraints.

In standard ERL, a "tug-of-war" emerges between exploration and safety. While the RL learner attempts to balance rewards and constraints, the Evolutionary Algorithm (EA) typically selects actors based solely on reward. This leads to "reckless" actors generating experiences that violate safety protocols, such as torque limits in robotic motors. This paper introduces Evolutionary Constrained Reinforcement Learning (ECRL), a hybrid framework designed to harmonize these components and ensure strict adherence to constraints without compromising performance.

Methodology

ECRL integrates evolutionary and gradient-based learning through three primary innovations designed to align the objectives of the EA and RL components.

The ECRL architecture integrates a population-based EA with an RL learner, utilizing stochastic ranking and a constraint buffer to manage safety requirements.
  1. Stochastic Ranking: To resolve the conflict in actor selection, ECRL employs a probabilistic ranking system. Individuals are compared based on constraint violations; if both are "safe," they are ranked by reward. If they are "unsafe," a probability parameter ($P_f$) determines whether to prioritize reward or the magnitude of the violation. This ensures the population evolves toward safety.
  2. Lagrangian Relaxation with Individual Multipliers: The framework treats the constrained problem as an unconstrained one by adding a penalty term. Uniquely, each actor maintains its own Lagrange multiplier ($\lambda$), facilitating a diverse range of risk-taking behaviors across the population.
  3. Constraint Buffer ($B_C$): To stabilize penalty updates, ECRL introduces a dedicated buffer for historical episodic constraints. This allows the RL learner to update penalty coefficients based on a stable "memory" of performance rather than noisy, recent data.

Experimental Setup

The framework was evaluated using the MuJoCo physics engine across five standard robotic control tasks: Ant, HalfCheetah, Walker2d, Hopper, and Swimmer. The primary constraint was a torque limit (average torque $\le 0.4$), simulating hardware limitations where excessive force can lead to mechanical failure.

Results and Discussion

ECRL was compared against state-of-the-art constrained RL methods, including Reward Constrained Policy Optimization (RCPO) and Interior-point Policy Optimization (IPO).

Performance comparison across MuJoCo tasks showing ECRL's superior reward achievement and constraint satisfaction compared to baselines.

The results demonstrate that ECRL consistently outperforms existing methods. In the Hopper and Swimmer tasks, ECRL was the only algorithm to successfully maintain the policy within torque constraints. In the Ant task, ECRL achieved significantly higher rewards (approximately 6000) compared to competitors (approximately 4000) while still satisfying constraints. Ablation studies further confirmed that both stochastic ranking and the constraint buffer are essential; removing either component led to a failure in balancing safety and reward.

Conclusion

ECRL effectively bridges the gap between evolutionary exploration and constrained reinforcement learning. By aligning the selection criteria of the EA with the safety requirements of the RL learner, the framework achieves superior performance in complex robotic benchmarks. Future work will explore ECRL's application in multi-constraint real-world scenarios and investigate methods to tighten constraint satisfaction in high-dimensional state spaces.

Summary updated

Abstract

Evolutionary algorithms have been used to evolve a population of actors to generate diverse experiences for training reinforcement learning agents, which helps to tackle the temporal credit assignment problem and improves the exploration efficiency. However, when adapting this approach to address constrained problems, balancing the trade-off between the reward and constraint violation is hard. In this paper, we propose a novel evolutionary constrained reinforcement learning (ECRL) algorithm, which adaptively balances the reward and constraint violation with stochastic ranking, and at the same time, restricts the policy's behaviour by maintaining a set of Lagrange relaxation coefficients with a constraint buffer. Extensive experiments on robotic control benchmarks show that our ECRL achieves outstanding performance compared to state-of-the-art algorithms. Ablation analysis shows the benefits of introducing stochastic ranking and constraint buffer.
BibTeX
@inproceedings{hu2023evolving,
  author = {Chengpeng Hu and Jiyuan Pei and Jialin Liu and Xin Yao},
  title = {{Evolving Constrained Reinforcement Learning Policy}},
  booktitle = {IJCNN},
  year = {2023},
  doi = {10.1109/ijcnn54540.2023.10191982},
}
Figure