Robust Reinforcement Learning using Least Squares Policy Iteration with Provable Performance Guarantees

Kishan Panaganti Badrinath, Dileep KalathilICML 2021 · OpenReview

Overview

Reinforcement Learning (RL) agents are typically trained in simulated environments. However, simulators are rarely perfect replicas of the physical world. A robot trained on a "perfect" digital floor might fail on a real-world carpet due to differences in friction, or a drone might crash because of unexpected wind conditions. This discrepancy is widely known as the sim-to-real gap.

To address this, researchers utilize the Robust Markov Decision Process (RMDP) framework, which seeks to find policies that perform well across a whole set of possible environments (an "uncertainty set"). While this works well for small, tabular problems, scaling it to complex, large-scale systems is notoriously difficult. Most current solutions for large systems rely on deep learning. While effective in practice, deep RL lacks the mathematical "safety net" or provable guarantees that engineers require for mission-critical applications.

This paper introduces Robust Least Squares Policy Iteration (RLSPI), the first model-free reinforcement learning algorithm that scales to large state spaces while providing formal performance guarantees. By combining linear function approximation with a novel robust policy evaluation technique, the authors create a system that remains stable even when the real-world environment differs significantly from the training simulator.

Methodology

The authors propose two main algorithmic components to achieve scalable, provable robustness:

  1. Robust Least Squares Policy Evaluation (RLSPE($\lambda$)): This is a multi-step, online algorithm that estimates the value of a current policy. The primary mathematical hurdle here is that robust Bellman operators—the equations used to update value estimates—are inherently non-linear because they must account for worst-case scenarios. The authors overcome this by defining a cleverly approximated robust operator that is mathematically tractable for least-squares methods while still converging to the correct robust value.
  2. Robust Least Squares Policy Iteration (RLSPI): This serves as the higher-level control loop. It uses RLSPE to evaluate a policy and then updates that policy to be "greedy" with respect to the newly calculated robust values.

Unlike contemporary deep RL approaches, this method relies on linear function approximation. While inherently less flexible than a deep neural network, linearity is treated as a feature rather than a bug. It allows the authors to leverage stochastic approximation theory to prove that the algorithm will converge, providing a specific mathematical bound on how close the final policy will be to the theoretical optimum.

Experimental Setup

The researchers evaluated their algorithm using standard continuous-control benchmarks from the OpenAI Gym library:

  • CartPole: Balancing a pole on a moving cart.
  • MountainCar: Driving an underpowered car up a steep hill.
  • Acrobot: Swinging a two-link pendulum to a target height.

To rigorously test robustness, the environments were intentionally perturbed. The team altered physical parameters like gravity, pole length, and external force magnitude, and introduced action noise (where the agent occasionally takes a random action instead of its intended one).

Key Results

The RLSPI algorithm consistently outperformed standard, non-robust baselines (such as LSPI and Q-Learning) when the environment parameters shifted away from their nominal training values.

Robustness to Action Perturbations

In the CartPole and Acrobot tasks, as the probability of "random actions" increased, RLSPI maintained a much higher average reward compared to non-robust methods. It successfully mitigated the impact of unexpected stochasticity in the agent's control mechanisms.

Robustness against random actions across CartPole, MountainCar, and Acrobot environments.

Stability Under Parameter Shifts

When physical parameters like gravity or force magnitude were altered by up to 80% from their training values, RLSPI's performance remained remarkably stable. In contrast, the performance of standard LSPI often plummeted under the same conditions.

Performance stability under parameter shifts (force magnitude, gravity, and length).

Interestingly, RLSPI performed comparably to "Soft-Robust" deep RL methods (which utilize complex neural networks) but with the critical added benefit of possessing mathematical proofs of performance that deep methods currently lack.

Limitations and Future Work

While RLSPI represents a significant step forward for provably safe RL, it has limitations:

  • Linear Constraints: The current proof and algorithm rely heavily on linear function approximation. While this scales to large state spaces, it may struggle with extremely complex, high-dimensional sensory data (like raw video feeds) where deep learning traditionally excels.
  • Uncertainty Set Choice: The experiments primarily utilized spherical uncertainty sets. The authors note that the choice of the uncertainty set is crucial and heavily influences the "conservativeness" of the resulting policy.

For future work, the authors plan to extend these theoretical results to non-linear function architectures and explore "offline" robust RL, where the agent learns safely from a fixed, pre-collected dataset rather than through live, potentially risky interaction.

Summary updated

Abstract

This paper addresses the problem of model-free reinforcement learning for Robust Markov Decision Process (RMDP) with large state spaces. The goal of the RMDPs framework is to find a policy that is ...
BibTeX
@inproceedings{badrinath2021robust,
  author = {Kishan Panaganti Badrinath and Dileep Kalathil},
  title = {{Robust Reinforcement Learning using Least Squares Policy Iteration with Provable Performance Guarantees}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=n7NKrNEtKEK},
}
Figure