Abstract
One major obstacle that precludes the success of reinforcement learning in real-world applications is the lack of robustness, either to model uncertainties or external disturbances, of the trained policies. Robustness is critical when the policies are trained in simulations instead of real world environment. In this work, we propose a risk-aware algorithm to learn robust policies in order to bridge the gap between simulation training and real-world implementation. Our algorithm is based on recently discovered distributional RL framework. We incorporate CVaR risk measure in sample based distributional policy gradients (SDPG) for learning risk-averse policies to achieve robustness against a range of system disturbances. We validate the robustness of risk-aware SDPG on multiple environments.
Introduction
Reinforcement Learning (RL) has achieved remarkable success in simulated environments, yet transitioning these policies to real-world applications remains a significant challenge. This "reality gap" stems from the fact that simulators are imperfect models of physical reality, often lacking the unpredictable disturbances—such as sensor noise, friction variations, or external forces—present in actual environments.
Standard RL algorithms typically focus on maximizing the expected return. While effective on average, this approach ignores the "long tail" of catastrophic outcomes. A policy that performs well in 95% of cases but fails completely in the remaining 5% is often unacceptable for real-world deployment. To address this, we propose a risk-aware framework that prioritizes avoiding worst-case scenarios over simple mean-performance maximization.
Methodology: Risk-Averse SDPG
Our approach integrates Distributional Reinforcement Learning (DRL) with formal Risk Measures to produce robust policies.
Distributional RL and CVaR
Unlike traditional RL, which estimates a single scalar value for an action, DRL learns the entire probability distribution of potential future rewards. We utilize Sample-based Distributional Policy Gradients (SDPG), representing the reward distribution through a set of samples. This flexibility allows the agent to model complex, multi-modal distributions without assuming a specific functional form (e.g., Gaussian).
To instill risk-aversion, we incorporate the Conditional Value at Risk (CVaR). CVaR measures the expected return of the worst $\alpha$-percentile of outcomes. By adjusting $\alpha$, we control the agent's level of caution: $\alpha=0$ corresponds to standard risk-neutral RL, while higher values force the agent to optimize for the "unlucky" tail of the distribution.
The Algorithm
In the Risk-Averse SDPG framework, the Critic network learns to generate samples representing the full distribution of future returns, optimized via a Quantile Huber Loss. The Actor network is then updated using the gradient of the CVaR of this distribution, as shown in the architecture diagram above. This sample-based approach allows the agent to understand and mitigate risk by directly observing the distribution of potential outcomes.
Experimental Setup
We evaluated our algorithm on three continuous control benchmarks from OpenAI Gym: BipedalWalker-v2, HalfCheetah-v2, and Walker2d-v2. To simulate the reality gap, we introduced Gaussian noise to the action forces during evaluation, testing robustness across four noise levels (0, 0.5, 1.0, and 1.5).
Results and Discussion
The experiments reveal a clear trade-off between peak performance in noise-free environments and stability under disturbance.
Key Findings
- Enhanced Robustness: Risk-averse agents ($\alpha > 0$) consistently maintained higher rewards as noise levels increased, whereas the performance of risk-neutral agents ($\alpha = 0$) degraded significantly.
- Environment-Specific Sensitivity: In BipedalWalker-v2, risk-averse policies outperformed the baseline across all noise levels. In Walker2d-v2, high risk-aversion ($\alpha = 0.5$) provided the most stability. Conversely, for HalfCheetah-v2, a moderate level of risk-aversion ($\alpha = 0.1$) achieved the best balance.
- Pessimism as a Stabilizer: By training the agent to handle the "worst-case" samples, the resulting policies learn to avoid unstable states where small perturbations could lead to failure (e.g., a robot falling over).
Conclusion
By leveraging the full reward distribution provided by DRL and optimizing for CVaR, we can train agents that are significantly more robust to external disturbances. This risk-aware approach offers a promising path toward bridging the reality gap in robotic applications. Future work will explore dynamic adjustment of risk levels and the application of other utility-based risk measures within this sample-based framework.
Summary updated