Introduction
In safety-critical domains such as autonomous driving or power grid management, Reinforcement Learning (RL) agents must adhere to strict safety constraints. While standard Constrained RL (CRL) focuses on satisfying constraints on average, Risk-Averse CRL (RaCRL) aims to minimize the risk of catastrophic failures using metrics like Conditional Value at Risk (CVaR). However, a significant challenge arises: risk-aversion often leads to extreme conservatism during training. Agents become "scared" of unknown regions, failing to explore paths that might be both safer and more rewarding in the long run.
The Conservatism Trap
The "conservatism trap" occurs when an agent overestimates the risk of unexplored areas. Because it prioritizes avoiding worst-case costs, it stays within a narrow, known "safe" zone, even if that zone is sub-optimal.
As shown in the figure above, standard risk-averse agents (blue) may fail to navigate complex environments because they perceive the uncertainty of a new path as an unacceptable risk. In contrast, the proposed ORAC algorithm successfully finds the optimal long-term path by being optimistic about unknown regions.
ORAC: Optimistic Risk-Averse Actor-Critic
To overcome this trap, the authors introduce ORAC, which integrates "Optimism in the Face of Uncertainty" into the Worst-Case Soft Actor-Critic (WCSAC) framework. ORAC's strategy is to be optimistic about what it doesn't know during the training phase to ensure a robust, safe policy for deployment.
Key Methodological Components
- Dual Confidence Bounds: ORAC calculates a Local Upper Confidence Bound (UCB) for rewards and a Local Lower Confidence Bound (LCB) for costs. This encourages the agent to "hope" that unknown areas have higher rewards and lower risks than currently estimated.
- Ensemble of Critics: The algorithm uses an ensemble of multiple cost critics to estimate epistemic uncertainty (uncertainty due to lack of data).
- Optimistic Policy Construction: ORAC maintains a separate exploratory policy that shifts action distributions toward regions maximizing the reward UCB and minimizing the cost LCB.
- Dynamic Lagrangian Weighting: A modified multiplier ($\bar{\lambda}$) reduces the cost penalty when optimistic cost estimates are below the safety threshold, further facilitating exploration.
Experimental Evaluation
The researchers evaluated ORAC against state-of-the-art baselines like WCSAC and SAC-Lagrangian across several benchmarks, including GuardedMaze, Safety-Gymnasium, and CityLearn.
Performance Highlights
- GuardedMaze: ORAC achieved a 100% success rate in finding the optimal risk-averse path in the most challenging settings, whereas WCSAC only succeeded 60% of the time.
- Safety-Gymnasium: As seen in the training curves, ORAC demonstrates superior sample efficiency. It finds high-reward policies significantly earlier than baselines while keeping the mean episode cost within the required threshold.
- CityLearn: In a high-fidelity energy management simulation, ORAC improved mean rewards and reduced critical safety violations (Outage Temperature Violations) from 0.70 to 0.65, proving its effectiveness in complex, real-world scenarios.
Conclusion and Future Work
ORAC demonstrates that optimism during training is a powerful tool for achieving risk-aversion at deployment. By distinguishing between aleatoric (inherent) and epistemic (knowledge-based) uncertainty, ORAC avoids the conservatism trap and discovers safer, more efficient policies.
A primary limitation is that optimistic exploration is inherently "unsafe" during the training phase itself, making it more suitable for simulations than for direct training on physical hardware. Future research may focus on adaptive mechanisms that balance optimism with real-time safety constraints to enable safer on-robot learning.
Summary updated