Abstract: Constrained reinforcement learning (RL) is an area of RL whose objective is to find an optimal policy that maximizes expected cumulative return while satisfying a given constraint. Most of the previous constrained RL works consider expected cumulative sum cost as the constraint. However, optimization with this constraint cannot guarantee a target probability of outage event that the cumulative sum cost exceeds a given threshold. This paper proposes a framework, named Quantile Constrained RL (QCRL), to constrain the quantile of the distribution of the cumulative sum cost that is a necessary and sufficient condition to satisfy the outage constraint. This is the first work that tackles the issue of applying the policy gradient theorem to the quantile and provides theoretical results for approximating the gradient of the quantile. Based on the derived theoretical results and the technique of the Lagrange multiplier, we construct a constrained RL algorithm named Quantile Constrained Policy Optimization (QCPO). We use distributional RL with the Large Deviation Principle (LDP) to estimate quantiles and tail probability of the cumulative sum cost for the implementation of QCPO. The implemented algorithm satisfies the outage probability constraint after the training period.
Introduction
Standard Constrained Reinforcement Learning (CRL) typically focuses on the Expected Cumulative Cost (ExpCP). In this paradigm, the agent ensures that, on average, it adheres to safety constraints. However, in high-stakes environments such as autonomous driving or telecommunications, average safety is insufficient. These domains require a guarantee against "outage events"—rare but catastrophic instances where the cumulative cost exceeds a critical threshold in a single run.
This paper addresses this gap by introducing Quantile Constrained Reinforcement Learning (QCRL). Instead of constraining the mean, QCRL constrains the quantile of the cost distribution, providing a direct mathematical guarantee for the outage probability.
The Outage Probability Constraint
The fundamental limitation of expected cost constraints is their inability to account for the "tail" of the cost distribution. An agent might have a low average cost but still suffer from a high probability of extreme failure. The authors demonstrate that constraining the $(1-\epsilon_0)$-quantile of the cost distribution to be less than a threshold $d_{th}$ is both a necessary and sufficient condition to ensure the probability of an outage (cost $> d_{th}$) remains below $\epsilon_0$.
As shown in the figure above, the quantile $q_{1-\epsilon_0}^\pi$ acts as the boundary. If this quantile is kept below the threshold $d_{th}$, the area under the curve representing the outage probability (Area = $\epsilon_0$) is strictly controlled.
Methodology: Quantile Constrained Policy Optimization (QCPO)
The primary theoretical hurdle in QCRL is that the standard Policy Gradient Theorem applies to expectations, whereas quantiles are non-linear operators. The authors overcome this by deriving a novel Temporal-Difference (TD) relation for quantiles, proving they can be approximated as the expectation of costs plus a policy-dependent additional cost term. This allows the use of state-based policy gradients, which are significantly more sample-efficient than previous trajectory-based methods.
Tail Modeling with the Large Deviation Principle (LDP)
To accurately estimate the probability of rare events, the authors utilize the Large Deviation Principle (LDP). Unlike prior methods that assume Gaussian cost distributions—which often decay too quickly and underestimate risk—QCPO fits a Weibull distribution to the tail of the cost distribution. This allows for more precise modeling of the actual decay rate of failure probabilities.
As summarized in the table above, QCPO is the first framework to combine theoretical rigor (state-based policy gradients) with deep RL implementation and advanced tail modeling (LDP with Weibull).
Experimental Results
The authors evaluated QCPO on several continuous control tasks, including SimpleButtonEnv, DynamicEnv, and GremlinEnv from Safety Gym and MuJoCo. Key findings include:
- Precision: QCPO maintained the outage probability almost exactly at the target levels (e.g., 0.1 or 0.2), whereas standard Lagrangian PPO often exceeded these limits by nearly double.
- Efficiency and Return: Compared to WCSAC, which uses a more conservative CVaR constraint and Gaussian modeling, QCPO achieved higher average returns. By being more precise about the safety boundary, QCPO avoids over-conservatism, allowing the agent to explore higher-reward strategies within the safety budget.
- Robustness: In complex environments like
GremlinEnv, QCPO was the only algorithm to consistently satisfy the outage probability constraint across multiple random seeds.
Conclusion
Quantile Constrained Reinforcement Learning provides a robust framework for safety-critical RL. By bridging the gap between policy gradient theorems and quantile constraints, and employing LDP for accurate tail modeling, QCPO offers a scalable solution for agents that must strictly limit the probability of failure. Future work may extend these theoretical results to environments with highly stochastic state transitions and further optimize the computational overhead of distributional tail fitting.
Summary updated