Robust Reinforcement Learning using Least Squares Policy Iteration with Provable Performance Guarantees

Kishan Panaganti, Dileep Kalathil

Abstract

This paper addresses the problem of model-free reinforcement learning for Robust Markov Decision Process (RMDP) with large state spaces. The goal of the RMDP framework is to find a policy that is robust against the parameter uncertainties due to the mismatch between the simulator model and real-world settings. We first propose the Robust Least Squares Policy Evaluation algorithm, which is a multi-step online model-free learning algorithm for policy evaluation. We prove the convergence of this algorithm using stochastic approximation techniques. We then propose Robust Least Squares Policy Iteration (RLSPI) algorithm for learning the optimal robust policy. We also give a general weighted Euclidean norm bound on the error (closeness to optimality) of the resulting policy. Finally, we demonstrate the performance of our RLSPI algorithm on some standard benchmark problems.

Introduction

Reinforcement Learning (RL) agents are typically trained in simulators, yet the real world rarely matches these simulations perfectly. Discrepancies in physical parameters—such as friction, mass, or sensor noise—often referred to as parameter uncertainty, can cause policies that performed optimally in training to fail during deployment. This is known as the "sim-to-real" gap.

The Robust Markov Decision Process (RMDP) framework addresses this by seeking a policy that performs well even in the "worst-case" scenario within a defined set of environment variations. Historically, robust RL methods have struggled to balance three competing needs: provable theoretical guarantees, scalability to large state spaces, and model-free operation. This paper introduces Robust Least Squares Policy Iteration (RLSPI), a method that achieves all three by combining robust optimization with linear function approximation.

Methodology

The authors propose a two-stage approach to robust learning:

  1. Robust Least Squares Policy Evaluation (RLSPE($\lambda$)): This multi-step, online algorithm estimates the value of a specific policy. Because the Robust Bellman Operator is inherently nonlinear, standard linear solvers cannot be used directly. The authors introduce a novel approximate operator that maintains a contraction mapping, allowing for proven convergence via stochastic approximation.
  2. Robust Least Squares Policy Iteration (RLSPI): Building on the evaluation step, RLSPI iteratively improves the policy using linear function approximation. By representing the value function as a weighted sum of features, the algorithm scales to continuous or large state spaces that are inaccessible to tabular methods.

Crucially, the authors provide a weighted Euclidean norm bound on the approximation error, mathematically limiting how far the learned policy can deviate from the true optimal robust policy.

Experimental Evaluation

The RLSPI algorithm was evaluated on standard OpenAI Gym benchmarks, including CartPole, MountainCar, and Acrobot. The experiments focused on two types of robustness: parameter shifts and action noise.

Resilience to Parameter Uncertainty

In the CartPole environment, the researchers tested the agents against shifts in physical constants like gravity, pole length, and force magnitude. While standard algorithms like LSPI and Q-Learning saw performance collapse as parameters moved away from their training values, RLSPI remained remarkably stable.

Robustness of RLSPI across varying physical parameters (force magnitude, gravity, and pole length) in the CartPole environment.

Robustness to Action Noise

Real-world disturbances were also modeled as "action perturbations," where an agent's intended action is occasionally replaced by a random one. As the probability of these noisy actions increased, RLSPI maintained significantly higher rewards compared to non-robust baselines.

Performance comparison under increasing action noise (probability of random actions).

Key Insights and Conclusion

A primary contribution of this work is demonstrating that approximation is the key to handling the nonlinearity of robust operators. By creating an approximation amenable to online learning, the authors bypassed the computational bottlenecks of previous robust methods. Furthermore, RLSPI achieved performance competitive with deep neural network-based methods (like Soft-Robust DQN) while maintaining the theoretical guarantees that deep RL lacks.

Future work will look toward extending these theoretical guarantees to nonlinear architectures like deep neural networks and adapting the algorithm for the "offline" setting, where agents must learn from fixed datasets without further environment interaction.

Summary updated

Abstract

This paper addresses the problem of model-free reinforcement learning for Robust Markov Decision Process (RMDP) with large state spaces. The goal of the RMDP framework is to find a policy that is robust against the parameter uncertainties due to the mismatch between the simulator model and real-world settings. We first propose the Robust Least Squares Policy Evaluation algorithm, which is a multi-step online model-free learning algorithm for policy evaluation. We prove the convergence of this algorithm using stochastic approximation techniques. We then propose Robust Least Squares Policy Iteration (RLSPI) algorithm for learning the optimal robust policy. We also give a general weighted Euclidean norm bound on the error (closeness to optimality) of the resulting policy. Finally, we demonstrate the performance of our RLSPI algorithm on some standard benchmark problems.
BibTeX
@misc{panaganti2020robust,
  author = {Kishan Panaganti and Dileep Kalathil},
  title = {{Robust Reinforcement Learning using Least Squares Policy Iteration with Provable Performance Guarantees}},
  year = {2020},
  eprint = {2006.11608},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.11608},
}
Figure