Variational Adversarial Training Towards Policies with Improved Robustness

Juncheng Dong, Hao-Lun Hsu, Qitong Gao, Vahid Tarokh, Miroslav PajicAISTATS 2025 · OpenReview

Abstract

Reinforcement learning (RL), while being the benchmark for policy formulation, often struggles to deliver robust solutions across varying scenarios, leading to marked performance drops under environmental perturbations. Traditional adversarial training, based on a two-player max-min game, is known to bolster the robustness of RL agents, but it faces challenges: first, the complexity of the worst-case optimization problem may induce over-optimism, and second, the choice of a specific set of potential adversaries might lead to over-pessimism by considering implausible scenarios. In this work, we first observe that these two challenges do not balance out each other. Thus, we propose to apply variational optimization to optimize over the worst-case distribution of the adversary instead of a single worst-case adversary. Moreover, to counteract over-optimism, we train the RL agent to maximize the lower quantile of the cumulative rewards under worst-case adversary distribution. Our novel algorithm demonstrates a significant advancement over existing robust RL methods, corroborating the importance of the identified challenges and the effectiveness of our approach. To alleviate computational overhead associated with the proposed approach, we also propose a simplified version with lower computational burden and only minimal performance degradation. Extensive experiments validate that our approaches consistently yield policies with superior robustness.

Introduction

Reinforcement Learning (RL) agents are notoriously brittle; a policy that works perfectly in a simulator might fail in the real world due to slight changes in friction, mass, or external disturbances. To fix this, researchers often use Adversarial Training, formulated as a max-min game: the agent tries to maximize rewards while an "adversary" tries to minimize them.

However, this approach faces two major flaws:

  1. Over-optimism: Finding the absolute "worst-case" adversary is a hard optimization problem. If the training process fails to find the true worst case, the agent becomes over-confident.
  2. Over-pessimism: If the adversary is allowed to be too strong or unrealistic, the agent becomes overly conservative, sacrificing performance in normal scenarios to protect against impossible ones.

This paper introduces ROSE (Robust Reinforcement Learning with Structured Adversarial Ensemble), a framework that trains agents against a diverse distribution of adversaries to achieve state-of-the-art robustness.

Methodology: The ROSE Framework

The authors propose Variational Adversarial Training (VAT) implemented through the ROSE algorithm. The key innovations include:

Conceptual overview of ROSE optimizing over a distribution of adversaries.
  • Variational Adversarial Ensemble: Instead of training against a single adversary, ROSE maintains an ensemble. It uses Stein Variational Policy Gradient (SVPG) to ensure this ensemble represents a diverse distribution of "bad" scenarios rather than collapsing into a single point.
  • Quantile Optimization (CVaR): To combat over-optimism, the agent optimizes for the Conditional Value at Risk (CVaR)—specifically, it focuses on maximizing the rewards in the lower $\alpha$-quantile (the worst-performing scenarios) of the adversary distribution.
  • ROSE-S (Simplified): Recognizing that variational optimization is computationally expensive, the authors introduced a simplified version that updates adversaries independently. This allows for parallel training while maintaining most of the robustness gains.

Experimental Setup

The researchers evaluated their methods using the MuJoCo physics engine (Ant, Hopper, Half-Cheetah, Walker2d, and InvertedPendulum). The agents were tested against three types of disturbances:

  1. Action Noise: Random perturbations to the agent's actions.
  2. Learned Adversaries: Hostile policies specifically trained to defeat the agent.
  3. Environmental Changes: Systematic variations in physical parameters like mass and friction.

Key Results

The ROSE framework consistently outperformed existing state-of-the-art robust RL methods like RARL and RAP:

  • Superior Adversarial Defense: In the Walker2d task under attack, ROSE achieved a normalized return of 0.70, compared to just 0.36 for a standard baseline and 0.34 for RARL.
  • Broad Generalization: Heatmaps of performance across different mass and friction settings show that ROSE maintains "high-reward" zones (red areas) across a much wider range of physical parameters than its competitors.
Performance heatmap of ROSE across varying mass and friction coefficients.
  • Efficiency: ROSE-S reduced the computational overhead of the variational approach by roughly 35% while still outperforming traditional baselines.

Conclusion

By shifting from a single worst-case adversary to a structured distribution, ROSE effectively balances the trade-offs between over-optimism and over-pessimism. The use of SVPG ensures a diverse defense, while CVaR optimization provides a stable performance floor. Future work will look to integrate robustness to observation noise and sensor errors into the framework.

Summary updated

Abstract

Reinforcement learning (RL), while being the benchmark for policy formulation, often struggles to deliver robust solutions across varying scenarios, leading to marked performance drops under environmental perturbations.~Traditional adversarial training, based on a two-player max-min game, is known to bolster the robustness of RL agents, but it faces challenges: first, the complexity of the worst-case optimization problem may induce *over-optimism*, and second, the choice of a specific set of potential adversaries might lead to *over-pessimism* by considering implausible scenarios. In this work, we first observe that these two challenges do not balance out each other. Thus, we propose to apply variational optimization to optimize over the worst-case distribution of the adversary instead of a single worst-case adversary. Moreover, to counteract over-optimism, we train the RL agent to maximize the lower quantile of the cumulative rewards under worst-case adversary distribution. Our novel algorithm demonstrates a significant advancement over existing robust RL methods, corroborating the importance of the identified challenges and the effectiveness of our approach. To alleviate computational overhead associated with the proposed approach, we also propose a simplified version with lower computational burden and only minimal performance degradation. Extensive experiments validate that our approaches consistently yield policies with superior robustness.
BibTeX
@inproceedings{dong2025variational,
  author = {Juncheng Dong and Hao-Lun Hsu and Qitong Gao and Vahid Tarokh and Miroslav Pajic},
  title = {{Variational Adversarial Training Towards Policies with Improved Robustness}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2025},
  url = {https://openreview.net/forum?id=zpfWwq0Ytl},
}
Figure