Introduction
Reinforcement Learning (RL) has demonstrated significant potential in solving complex control tasks. However, it often struggles with efficient exploration and "temporal credit assignment"—the challenge of attributing rewards to specific past actions. Evolutionary Reinforcement Learning (ERL) addresses these issues by maintaining a population of diverse actors. Yet, applying ERL to real-world scenarios introduces a new challenge: safety constraints.
In standard ERL, a "tug-of-war" emerges between exploration and safety. While the RL learner attempts to balance rewards and constraints, the Evolutionary Algorithm (EA) typically selects actors based solely on reward. This leads to "reckless" actors generating experiences that violate safety protocols, such as torque limits in robotic motors. This paper introduces Evolutionary Constrained Reinforcement Learning (ECRL), a hybrid framework designed to harmonize these components and ensure strict adherence to constraints without compromising performance.
Methodology
ECRL integrates evolutionary and gradient-based learning through three primary innovations designed to align the objectives of the EA and RL components.
- Stochastic Ranking: To resolve the conflict in actor selection, ECRL employs a probabilistic ranking system. Individuals are compared based on constraint violations; if both are "safe," they are ranked by reward. If they are "unsafe," a probability parameter ($P_f$) determines whether to prioritize reward or the magnitude of the violation. This ensures the population evolves toward safety.
- Lagrangian Relaxation with Individual Multipliers: The framework treats the constrained problem as an unconstrained one by adding a penalty term. Uniquely, each actor maintains its own Lagrange multiplier ($\lambda$), facilitating a diverse range of risk-taking behaviors across the population.
- Constraint Buffer ($B_C$): To stabilize penalty updates, ECRL introduces a dedicated buffer for historical episodic constraints. This allows the RL learner to update penalty coefficients based on a stable "memory" of performance rather than noisy, recent data.
Experimental Setup
The framework was evaluated using the MuJoCo physics engine across five standard robotic control tasks: Ant, HalfCheetah, Walker2d, Hopper, and Swimmer. The primary constraint was a torque limit (average torque $\le 0.4$), simulating hardware limitations where excessive force can lead to mechanical failure.
Results and Discussion
ECRL was compared against state-of-the-art constrained RL methods, including Reward Constrained Policy Optimization (RCPO) and Interior-point Policy Optimization (IPO).
The results demonstrate that ECRL consistently outperforms existing methods. In the Hopper and Swimmer tasks, ECRL was the only algorithm to successfully maintain the policy within torque constraints. In the Ant task, ECRL achieved significantly higher rewards (approximately 6000) compared to competitors (approximately 4000) while still satisfying constraints. Ablation studies further confirmed that both stochastic ranking and the constraint buffer are essential; removing either component led to a failure in balancing safety and reward.
Conclusion
ECRL effectively bridges the gap between evolutionary exploration and constrained reinforcement learning. By aligning the selection criteria of the EA with the safety requirements of the RL learner, the framework achieves superior performance in complex robotic benchmarks. Future work will explore ECRL's application in multi-constraint real-world scenarios and investigate methods to tighten constraint satisfaction in high-dimensional state spaces.
Summary updated