Abstract
Deep reinforcement learning has recently made significant progress in solving computer games and robotic control tasks. A known problem, though, is that policies overfit to the training environment and may not avoid rare, catastrophic events such as automotive accidents. A classical technique for improving the robustness of reinforcement learning algorithms is to train on a set of randomized environments, but this approach only guards against common situations. Recently, robust adversarial reinforcement learning (RARL) was developed, which allows efficient applications of random and systematic perturbations by a trained adversary. A limitation of RARL is that only the expected control objective is optimized; there is no explicit modeling or optimization of risk. Thus the agents do not consider the probability of catastrophic events (i.e., those inducing abnormally large negative reward), except through their effect on the expected objective. In this paper we introduce risk-averse robust adversarial reinforcement learning (RARARL), using a risk-averse protagonist and a risk-seeking adversary. We test our approach on a self-driving vehicle controller. We use an ensemble of policy networks to model risk as the variance of value functions. We show through experiments that a risk-averse agent is better equipped to handle a risk-seeking adversary, and experiences substantially fewer crashes compared to agents trained without an adversary.
Introduction
Deep Reinforcement Learning (RL) has achieved remarkable success in complex domains, yet these agents often exhibit significant "brittleness" when deployed in real-world scenarios. Standard RL algorithms typically optimize for the expected reward, which represents the average outcome across training episodes. This focus on the mean performance often ignores "tail-end" risks—rare but catastrophic events like a self-driving car encountering black ice or an unexpected obstacle. While Robust Adversarial Reinforcement Learning (RARL) improved robustness by introducing an adversary to perturb the environment, it still primarily optimized for the average case without explicit risk modeling. This paper introduces Risk Averse Robust Adversarial Reinforcement Learning (RARARL), a framework designed to quantify and mitigate risk by incorporating a risk-averse protagonist and a risk-seeking adversary.
Methodology: The RARARL Framework
The core of RARARL is a zero-sum competitive game between two agents: a risk-averse protagonist and a risk-seeking adversary.
Risk Modeling via Ensemble Methods
To quantify risk without a predefined model of the environment, the authors utilize an ensemble of $k$ Q-value networks. These networks share a common convolutional base but have independent output heads. Risk is defined as the variance of the Q-values across these $k$ models, representing the agent's "epistemic uncertainty" regarding specific states. The protagonist’s objective function is penalized by this variance, encouraging it to avoid states where the ensemble's predictions diverge. Conversely, the adversary’s objective is augmented by this variance, driving it to actively seek out and exploit regions of the state space where the protagonist is most uncertain.
Training Dynamics
The agents are trained sequentially: the protagonist takes $m$ steps to optimize its policy, followed by the adversary taking $n$ steps to apply perturbations. This iterative process forces the protagonist to develop a policy that is robust not just to random noise, but to targeted, worst-case scenarios identified by the risk-seeking adversary.
Experimental Setup
The framework was evaluated using TORCS (The Open Racing Car Simulator) on the Michigan Speedway track. The input consists of stacked 84x84x3 RGB images. The action space includes 9 discrete combinations of steering and acceleration/braking. A "catastrophe" is defined as a crash or the vehicle becoming stuck, resulting in a significant negative reward ($r_{cat} = -2.5$).
Results and Discussion
The RARARL agent (denoted as bsdqnadvriskaverse) demonstrated superior performance in avoiding catastrophic failures compared to vanilla DQN and standard RARL baselines.
As shown in the results, the risk-averse agent trained against an adversary achieved a significantly higher "Catastrophe Reward Per Episode" (closer to zero), indicating far fewer crashes. Specifically, under adversarial attack, the RARARL agent maintained a catastrophe reward of approximately -0.1, whereas the vanilla DQN dropped to -4.0.
The Adversary as a Stress Tester
A key insight is that the risk-seeking adversary acts as a targeted stress tester. Unlike random perturbations, which often cancel out over time, the trained adversary identifies specific "cracks" in the protagonist's policy. By forcing the protagonist into these high-variance regions during training, RARARL ensures the final policy is prepared for deliberate, multi-step challenges that standard agents often fail to navigate.
Conclusion and Future Work
RARARL provides a robust framework for safety-critical RL by explicitly modeling risk as value function variance. While the current implementation uses discrete actions in a simulated environment, it establishes a foundation for more complex, continuous control tasks. Future research will focus on sim-to-real transfer and applying this framework to other high-stakes domains, such as surgical robotics, where the cost of failure is prohibitively high.
Summary updated