Optimistic Exploration for Risk-Averse Constrained Reinforcement Learning

James McCarthy, Radu Marinescu, Elizabeth Daly, Ivana DusparicarXiv 2025 · DOI · arXiv

Introduction

In safety-critical domains such as autonomous driving or power grid management, Reinforcement Learning (RL) agents must adhere to strict safety constraints. While standard Constrained RL (CRL) focuses on satisfying constraints on average, Risk-Averse CRL (RaCRL) aims to minimize the risk of catastrophic failures using metrics like Conditional Value at Risk (CVaR). However, a significant challenge arises: risk-aversion often leads to extreme conservatism during training. Agents become "scared" of unknown regions, failing to explore paths that might be both safer and more rewarding in the long run.

The Conservatism Trap

The "conservatism trap" occurs when an agent overestimates the risk of unexplored areas. Because it prioritizes avoiding worst-case costs, it stays within a narrow, known "safe" zone, even if that zone is sub-optimal.

Visualization of the conservatism trap: Risk-averse control (blue) gets stuck, while ORAC (white) explores a longer but safer path.

As shown in the figure above, standard risk-averse agents (blue) may fail to navigate complex environments because they perceive the uncertainty of a new path as an unacceptable risk. In contrast, the proposed ORAC algorithm successfully finds the optimal long-term path by being optimistic about unknown regions.

ORAC: Optimistic Risk-Averse Actor-Critic

To overcome this trap, the authors introduce ORAC, which integrates "Optimism in the Face of Uncertainty" into the Worst-Case Soft Actor-Critic (WCSAC) framework. ORAC's strategy is to be optimistic about what it doesn't know during the training phase to ensure a robust, safe policy for deployment.

Key Methodological Components

  • Dual Confidence Bounds: ORAC calculates a Local Upper Confidence Bound (UCB) for rewards and a Local Lower Confidence Bound (LCB) for costs. This encourages the agent to "hope" that unknown areas have higher rewards and lower risks than currently estimated.
  • Ensemble of Critics: The algorithm uses an ensemble of multiple cost critics to estimate epistemic uncertainty (uncertainty due to lack of data).
  • Optimistic Policy Construction: ORAC maintains a separate exploratory policy that shifts action distributions toward regions maximizing the reward UCB and minimizing the cost LCB.
  • Dynamic Lagrangian Weighting: A modified multiplier ($\bar{\lambda}$) reduces the cost penalty when optimistic cost estimates are below the safety threshold, further facilitating exploration.

Experimental Evaluation

The researchers evaluated ORAC against state-of-the-art baselines like WCSAC and SAC-Lagrangian across several benchmarks, including GuardedMaze, Safety-Gymnasium, and CityLearn.

Training performance on Safety-Gymnasium tasks. ORAC (green) achieves higher rewards more quickly while maintaining safety costs below the threshold compared to WCSAC (orange).

Performance Highlights

  • GuardedMaze: ORAC achieved a 100% success rate in finding the optimal risk-averse path in the most challenging settings, whereas WCSAC only succeeded 60% of the time.
  • Safety-Gymnasium: As seen in the training curves, ORAC demonstrates superior sample efficiency. It finds high-reward policies significantly earlier than baselines while keeping the mean episode cost within the required threshold.
  • CityLearn: In a high-fidelity energy management simulation, ORAC improved mean rewards and reduced critical safety violations (Outage Temperature Violations) from 0.70 to 0.65, proving its effectiveness in complex, real-world scenarios.

Conclusion and Future Work

ORAC demonstrates that optimism during training is a powerful tool for achieving risk-aversion at deployment. By distinguishing between aleatoric (inherent) and epistemic (knowledge-based) uncertainty, ORAC avoids the conservatism trap and discovers safer, more efficient policies.

A primary limitation is that optimistic exploration is inherently "unsafe" during the training phase itself, making it more suitable for simulations than for direct training on physical hardware. Future research may focus on adaptive mechanisms that balance optimism with real-time safety constraints to enable safer on-robot learning.

Summary updated

Abstract

Risk-averse Constrained Reinforcement Learning (RaCRL) aims to learn policies that minimise the likelihood of rare and catastrophic constraint violations caused by an environment's inherent randomness. In general, risk-aversion leads to conservative exploration of the environment which typically results in converging to sub-optimal policies that fail to adequately maximise reward or, in some cases, fail to achieve the goal. In this paper, we propose an exploration-based approach for RaCRL called Optimistic Risk-averse Actor Critic (ORAC), which constructs an exploratory policy by maximising a local upper confidence bound of the state-action reward value function whilst minimising a local lower confidence bound of the risk-averse state-action cost value function. Specifically, at each step, the weighting assigned to the cost value is increased or decreased if it exceeds or falls below the safety constraint value. This way the policy is encouraged to explore uncertain regions of the environment to discover high reward states whilst still satisfying the safety constraints. Our experimental results demonstrate that the ORAC approach prevents convergence to sub-optimal policies and improves significantly the reward-cost trade-off in various continuous control tasks such as Safety-Gymnasium and a complex building energy management environment CityLearn.
BibTeX
@misc{mccarthy2025optimistic,
  author = {James McCarthy and Radu Marinescu and Elizabeth Daly and Ivana Dusparic},
  title = {{Optimistic Exploration for Risk-Averse Constrained Reinforcement Learning}},
  year = {2025},
  eprint = {2507.08793},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.08793},
}
Figure