Gradient-Adaptive Pareto Optimization for Constrained Reinforcement Learning

Zixian Zhou, Mengda Huang, Feiyang Pan, Jia He, Xiang Ao, Dandan Tu, Qing He

Abstract

Constrained Reinforcement Learning (CRL) requires agents to maximize rewards while strictly adhering to safety constraints. Traditional Multi-Objective Optimization (MOO) and Pareto methods often fail in CRL due to "Pareto traps"—biases toward shorter gradients and an inability to sacrifice rewards for feasibility. We introduce Gradient-Adaptive Pareto Optimization (GCPO), a framework that utilizes gradient rebalancing and perturbation to recalibrate updates. GCPO ensures a balanced trade-off between objectives and prioritizes safety when constraints are violated, outperforming existing Lagrangian and CPO methods on the SafetyGym benchmark.

Introduction

In Constrained Reinforcement Learning, an agent's objective is to maximize a cumulative reward signal $J_R(\pi)$ while ensuring that a cost signal $J_C(\pi)$ remains below a threshold $\zeta$. While this can be viewed as a multi-objective problem, standard Pareto optimization techniques frequently struggle in practice. The authors identify two primary failures, termed "Pareto traps":

  1. The Imbalance Trap: Standard solvers tend to favor the objective with the smaller gradient magnitude. In RL, this often leads to the agent focusing on the "easier" objective (e.g., reward) while neglecting safety, or vice versa.
  2. The Feasibility Trap: Pure Pareto methods aim to improve all objectives simultaneously. However, in CRL, it is often necessary to temporarily decrease rewards to pull the agent back into the feasible region ($J_C(\pi) \leq \zeta$).
Illustration of gradient synthesis and the bias toward shorter gradients.

Methodology: Gradient-Adaptive Pareto Optimization (GCPO)

GCPO builds upon the Proximal Policy Optimization (PPO) algorithm, introducing two novel recalibration layers to the gradient update process:

Gradient Rebalancing

To address the Imbalance Trap, GCPO scales the gradients of rewards and costs based on their magnitudes. By setting rebalanced gradients proportional to the reciprocal of their original length ($1/\|\nabla\|^2$), the optimization naturally prioritizes the "underdeveloped" objective—the one further from its local optimum. This prevents the agent from "lazily" following a small reward gradient while ignoring significant safety risks.

Gradient Perturbation

To overcome the Feasibility Trap, GCPO implements a "safety-first" override. When a cost constraint is violated, the algorithm applies a clipping threshold ($t$) to the reward gradient. This artificial limitation forces the optimization to prioritize cost reduction, even if it necessitates a temporary drop in rewards. This mechanism allows the agent to "backtrack" toward the feasible region, a behavior standard Pareto methods lack.

Experimental Evaluation

The researchers evaluated GCPO using the SafetyGym benchmark, utilizing the MuJoCo physics engine. The agent (a point-based robot) was tested on three tasks—Goal, Button, and Push—at two difficulty levels. Level 2 tasks featured increased obstacles, making safety constraints significantly harder to satisfy.

Comparison of optimization routes across different CRL methods. GCPO (d) demonstrates a more direct path to the feasible Pareto frontier compared to baselines.

Key Results

GCPO was compared against Lagrangian methods (TRPO-L, PPO-L) and Constrained Policy Optimization (CPO). Key findings include:

  • Robust Constraint Satisfaction: In high-difficulty Level 2 tasks, GCPO was often the only algorithm to consistently maintain cumulative costs below the required threshold.
  • Higher Reward Efficiency: While Lagrangian methods often became "too safe" and failed to explore high-reward regions, GCPO achieved significantly higher rewards. For instance, in the Goal-Lvl 2 task, GCPO reached a reward of 8.39, whereas PPO-Lagrangian achieved only 1.15 while maintaining similar safety levels.
  • Ablation Studies: Removing either the rebalancing or perturbation components led to significant performance degradation, either through constraint violation or a failure to maximize rewards, confirming that both mechanisms are essential for stable CRL.

Conclusion and Future Work

GCPO provides a robust framework for CRL by addressing the inherent biases of traditional Pareto optimization. By recalibrating gradients to account for objective imbalance and feasibility requirements, it achieves a superior balance between safety and performance. Future research will focus on mathematically formalizing the heuristic tuning of the perturbation threshold and extending the framework to handle high-dimensional agents and multiple competing constraints.

Summary updated

Abstract

Constrained Reinforcement Learning (CRL) burgeons broad interest in recent years, which pursues maximizing long-term returns while constraining costs. Although CRL can be cast as a multi-objective optimization problem, it is still facing the key challenge that gradient-based Pareto optimization methods tend to stick to known Pareto-optimal solutions even when they yield poor returns (e.g., the safest self-driving car that never moves) or violate the constraints (e.g., the record-breaking racer that crashes the car). In this paper, we propose Gradient-adaptive Constrained Policy Optimization (GCPO for short), a novel Pareto optimization method for CRL with two adaptive gradient recalibration techniques. First, to find Pareto-optimal solutions with balanced performance over all targets, we propose gradient rebalancing which forces the agent to improve more on under-optimized objectives at every policy iteration. Second, to guarantee that the cost constraints are satisfied, we propose gradient perturbation that can temporarily sacrifice the returns for costs. Experiments on the SafetyGym benchmarks show that our method consistently outperforms previous CRL methods in reward while satisfying the constraints.
BibTeX
@inproceedings{zhou2023gradient,
  author = {Zixian Zhou and Mengda Huang and Feiyang Pan and Jia He and Xiang Ao and Dandan Tu and Qing He},
  title = {{Gradient-Adaptive Pareto Optimization for Constrained Reinforcement Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i9.26353},
}
Figure