Abstract
Reinforcement learning has demonstrated impressive performance in various challenging problems such as robotics, board games, and classical arcade games. However, its real-world applications can be hindered by the absence of robustness and safety in the learned policies. More specifically, an RL agent that trains in a certain Markov decision process (MDP) often struggles to perform well in nearly identical MDPs. To address this issue, we employ the framework of Robust MDPs (RMDPs) in a model-based setting and introduce a novel learned transition model. Our method specifically incorporates an auxiliary pessimistic model, updated adversarially, to estimate the worst-case MDP within a Kullback-Leibler uncertainty set. In comparison to several existing works, our work does not impose any additional conditions on the training environment, such as the need for a parametric simulator. To test the effectiveness of the proposed pessimistic model in enhancing policy robustness, we integrate it into a practical RL algorithm, called Robust Model-Based Policy Optimization (RMBPO). Our experimental results indicate a notable improvement in policy robustness on high-dimensional MuJoCo control tasks, with the auxiliary model enhancing the performance of the learned policy in distorted MDPs. We further explore the learned deviation between the proposed auxiliary world model and the nominal model, to examine how pessimism is achieved. By learning a pessimistic world model and demonstrating its role in improving policy robustness, our research contributes towards making (model-based) RL more robust.
Introduction: The Challenge of Policy Robustness
Reinforcement Learning (RL) agents often exhibit a significant performance drop when transitioned from training simulators to real-world environments—a challenge known as the "sim-to-real gap." This fragility stems from the agent's tendency to overfit to the specific dynamics of its training Markov Decision Process (MDP). Even minor discrepancies in physical parameters, such as friction or mass, can lead to policy failure.
To mitigate this, the framework of Robust MDPs (RMDPs) considers a set of possible transition models rather than a single fixed one. However, traditional robust methods often rely on parametric simulators or the ability to reset environments to specific states, which are not always available. This paper introduces Robust Model-Based Policy Optimization (RMBPO), which achieves robustness in a model-based setting without these restrictive requirements.
Methodology: Adversarial Auxiliary Models
RMBPO builds upon the Model-Based Policy Optimization (MBPO) framework by introducing an Adversarial Auxiliary Model ($g_\psi$). The training process involves a zero-sum game between the agent's policy and this auxiliary model:
- Nominal Model Learning: The agent learns a standard transition model ($p_\theta$) to capture the expected dynamics of the environment.
- Pessimistic Perturbation: The auxiliary model ($g_\psi$) perturbs the nominal model's predictions. It is trained to minimize the agent's expected return while remaining within a defined Kullback-Leibler (KL) uncertainty set relative to the nominal model.
- Adversarial Training: The policy is optimized against this "worst-case" auxiliary model. This forces the agent to develop strategies that are resilient to the most damaging (yet realistic) deviations in environment dynamics.
Experimental Evaluation
The effectiveness of RMBPO was evaluated on high-dimensional continuous control tasks from the MuJoCo suite: InvertedPendulum-v2, Hopper-v3, and Walker2d-v3. To assess robustness, the agents were tested in environments with distorted physical parameters, including varied mass and friction coefficients.
Key Results
- Superior Robustness: RMBPO consistently outperformed the standard MBPO baseline across nearly all distorted evaluation environments.
- Survival Range: In the
InvertedPendulumtask, while the baseline's performance plummeted as pendulum mass increased, RMBPO maintained high returns over a significantly broader range of mass values. - Resilience to Noise: RMBPO demonstrated superior resilience to increased action noise during testing, suggesting the learned policies are less dependent on precise execution and more stable overall.
Discussion: The Nature of Learned Pessimism
Analysis of the auxiliary model reveals how it achieves pessimism. In the Hopper task, for instance, the adversarial model learns to increase the torso's angular velocity while decreasing leg velocity, effectively trying to trip the robot. By training against these targeted "attacks," the RMBPO agent learns a more stable, conservative gait that prioritizes balance over raw speed.
The hyperparameter $\eta$, which controls the size of the KL uncertainty set, plays a crucial role. A well-tuned $\eta$ balances the need for realism with the benefits of pessimism, allowing the agent to prepare for likely real-world variations without becoming overly cautious.
Conclusion
RMBPO provides a practical and effective approach to robust model-based RL. By leveraging an adversarial auxiliary model to simulate worst-case scenarios within a KL uncertainty set, it produces agents capable of handling significant environmental distortions. Crucially, this is achieved without requiring access to simulator internals, making it a promising step toward deploying robust RL in real-world robotics and control systems.
Summary updated