Improving Robustness via Risk Averse Distributional Reinforcement Learning

Rahul Singh, Qinsheng Zhang, Yongxin Chen

Abstract

One major obstacle that precludes the success of reinforcement learning in real-world applications is the lack of robustness, either to model uncertainties or external disturbances, of the trained policies. Robustness is critical when the policies are trained in simulations instead of real world environment. In this work, we propose a risk-aware algorithm to learn robust policies in order to bridge the gap between simulation training and real-world implementation. Our algorithm is based on recently discovered distributional RL framework. We incorporate CVaR risk measure in sample based distributional policy gradients (SDPG) for learning risk-averse policies to achieve robustness against a range of system disturbances. We validate the robustness of risk-aware SDPG on multiple environments.

Introduction

Reinforcement Learning (RL) has achieved remarkable success in simulated environments, yet transitioning these policies to real-world applications remains a significant challenge. This "reality gap" stems from the fact that simulators are imperfect models of physical reality, often lacking the unpredictable disturbances—such as sensor noise, friction variations, or external forces—present in actual environments.

Standard RL algorithms typically focus on maximizing the expected return. While effective on average, this approach ignores the "long tail" of catastrophic outcomes. A policy that performs well in 95% of cases but fails completely in the remaining 5% is often unacceptable for real-world deployment. To address this, we propose a risk-aware framework that prioritizes avoiding worst-case scenarios over simple mean-performance maximization.

Methodology: Risk-Averse SDPG

Our approach integrates Distributional Reinforcement Learning (DRL) with formal Risk Measures to produce robust policies.

Distributional RL and CVaR

Unlike traditional RL, which estimates a single scalar value for an action, DRL learns the entire probability distribution of potential future rewards. We utilize Sample-based Distributional Policy Gradients (SDPG), representing the reward distribution through a set of samples. This flexibility allows the agent to model complex, multi-modal distributions without assuming a specific functional form (e.g., Gaussian).

To instill risk-aversion, we incorporate the Conditional Value at Risk (CVaR). CVaR measures the expected return of the worst $\alpha$-percentile of outcomes. By adjusting $\alpha$, we control the agent's level of caution: $\alpha=0$ corresponds to standard risk-neutral RL, while higher values force the agent to optimize for the "unlucky" tail of the distribution.

Architecture of the Risk-Averse SDPG framework, illustrating the distributional critic and the actor gradient update rule.

The Algorithm

In the Risk-Averse SDPG framework, the Critic network learns to generate samples representing the full distribution of future returns, optimized via a Quantile Huber Loss. The Actor network is then updated using the gradient of the CVaR of this distribution, as shown in the architecture diagram above. This sample-based approach allows the agent to understand and mitigate risk by directly observing the distribution of potential outcomes.

Experimental Setup

We evaluated our algorithm on three continuous control benchmarks from OpenAI Gym: BipedalWalker-v2, HalfCheetah-v2, and Walker2d-v2. To simulate the reality gap, we introduced Gaussian noise to the action forces during evaluation, testing robustness across four noise levels (0, 0.5, 1.0, and 1.5).

Results and Discussion

The experiments reveal a clear trade-off between peak performance in noise-free environments and stability under disturbance.

Performance curves and reward distributions for BipedalWalker-v2 across different risk levels (alpha).

Key Findings

  • Enhanced Robustness: Risk-averse agents ($\alpha > 0$) consistently maintained higher rewards as noise levels increased, whereas the performance of risk-neutral agents ($\alpha = 0$) degraded significantly.
  • Environment-Specific Sensitivity: In BipedalWalker-v2, risk-averse policies outperformed the baseline across all noise levels. In Walker2d-v2, high risk-aversion ($\alpha = 0.5$) provided the most stability. Conversely, for HalfCheetah-v2, a moderate level of risk-aversion ($\alpha = 0.1$) achieved the best balance.
  • Pessimism as a Stabilizer: By training the agent to handle the "worst-case" samples, the resulting policies learn to avoid unstable states where small perturbations could lead to failure (e.g., a robot falling over).

Conclusion

By leveraging the full reward distribution provided by DRL and optimizing for CVaR, we can train agents that are significantly more robust to external disturbances. This risk-aware approach offers a promising path toward bridging the reality gap in robotic applications. Future work will explore dynamic adjustment of risk levels and the application of other utility-based risk measures within this sample-based framework.

Summary updated

Abstract

One major obstacle that precludes the success of reinforcement learning in real-world applications is the lack of robustness, either to model uncertainties or external disturbances, of the trained policies. Robustness is critical when the policies are trained in simulations instead of real world environment. In this work, we propose a risk-aware algorithm to learn robust policies in order to bridge the gap between simulation training and real-world implementation. Our algorithm is based on recently discovered distributional RL framework. We incorporate CVaR risk measure in sample based distributional policy gradients (SDPG) for learning risk-averse policies to achieve robustness against a range of system disturbances. We validate the robustness of risk-aware SDPG on multiple environments.
BibTeX
@misc{singh2020improving,
  author = {Rahul Singh and Qinsheng Zhang and Yongxin Chen},
  title = {{Improving Robustness via Risk Averse Distributional Reinforcement Learning}},
  year = {2020},
  eprint = {2005.00585},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2005.00585},
}
Figure