Abstract
Traditional reinforcement learning (RL) agents typically aim to maximize average expected rewards, which can lead to catastrophic failures in safety-critical environments. This paper proposes a risk-sensitive, actor-free RL framework that prioritizes safety by focusing on tail risks. By employing Partially Input-Convex Neural Networks (PICNNs), the framework eliminates the traditional actor network, instead determining optimal actions through direct convex optimization.
Introduction
In real-world applications such as robotics or industrial control, "risk-neutral" agents are often insufficient. A high-reward path that carries even a small risk of failure is unacceptable. Safe RL attempts to address this through constraints, but standard actor-critic methods frequently suffer from training instability and local minima. The "tug-of-war" between the actor (policy) and critic (value estimator) can lead to suboptimal performance and complex hyperparameter tuning.
Methodology
The authors introduce two primary innovations to overcome these challenges:
Risk Sensitivity via CVaR
Instead of maximizing expected return, the agent minimizes the Conditional Value at Risk (CVaR). This metric focuses on the "tail" of the cost distribution—the worst outcomes (e.g., the worst 10%). By modeling the distribution of future costs as a Gaussian, the authors derive a closed-form risk-sensitive objective that balances the mean and variance of costs.
Actor-free Architecture via PICNNs
To eliminate the actor network, the authors utilize Partially Input Convex Neural Networks (PICNNs). These networks are designed to be convex with respect to the action input while maintaining non-convexity with respect to the state.
Because the critic is convex in terms of actions, the optimal action for any given state can be found efficiently using standard gradient-based optimization (e.g., the Adam optimizer) during each interaction step. This ensures the policy is always perfectly aligned with the critic's current knowledge, bypassing the stability issues of actor-critic architectures.
Experimental Evaluation
The method was evaluated on a simulated cascade water tank level control task. The objective is to maintain a target water level in a lower tank by adjusting pump voltage. A "critical event" is defined as the upper tank overflowing (exceeding 10 cm). The reward function is based on the distance to the goal, while the cost function is tied to the upper tank's water level.
Results and Discussion
The proposed actor-free approach was compared against CVaR-TD3, a standard actor-critic method modified for risk sensitivity.
Stability and Convergence
As shown in the training plots, the actor-free approach demonstrated significantly lower variance during training. While CVaR-TD3 often became trapped in poor local minima, the PICNN-based method consistently identified globally optimal actions due to the convex nature of the optimization landscape.
Risk Control
Adjusting the risk parameter ($\alpha$) resulted in predictable behavioral shifts. At higher risk-aversion levels ($\alpha = 0.1$), the agent adopted a more conservative strategy, maintaining lower water levels in the upper tank to prevent overflow, even if it meant being slightly further from the target water level in the lower tank.
Conclusion
By transforming the policy learning problem into a convex optimization problem, this framework offers a robust alternative to traditional actor-critic methods in safe RL. While the current Gaussian assumption and the computational cost of per-step optimization are limitations, the approach provides a clear path toward more stable and safety-aware autonomous systems. Future work will explore non-Gaussian cost distributions to better capture heavy-tailed risks.
Summary updated