Abstract
This paper addresses the problem of model-free reinforcement learning for Robust Markov Decision Process (RMDP) with large state spaces. The goal of the RMDP framework is to find a policy that is robust against the parameter uncertainties due to the mismatch between the simulator model and real-world settings. We first propose the Robust Least Squares Policy Evaluation algorithm, which is a multi-step online model-free learning algorithm for policy evaluation. We prove the convergence of this algorithm using stochastic approximation techniques. We then propose Robust Least Squares Policy Iteration (RLSPI) algorithm for learning the optimal robust policy. We also give a general weighted Euclidean norm bound on the error (closeness to optimality) of the resulting policy. Finally, we demonstrate the performance of our RLSPI algorithm on some standard benchmark problems.
Introduction: Bridging the Sim-to-Real Gap
Reinforcement Learning (RL) agents are typically trained in simulators, but these environments are rarely perfect representations of reality. Discrepancies in physical parameters like friction, mass, or sensor noise—collectively known as the sim-to-real gap—can cause policies that excel in simulation to fail catastrophically in real-world applications.
The Robust MDP (RMDP) framework addresses this by seeking policies that perform well under the "worst-case" scenario within a defined set of environment parameters. However, existing robust RL methods often face a trade-off: tabular methods offer strong theoretical guarantees but fail to scale beyond tiny problems, while deep robust RL methods scale to complex tasks but lack formal performance proofs. This work introduces RLSPI, the first model-free RL algorithm for RMDPs that utilizes linear function approximation to handle large state spaces while providing rigorous mathematical guarantees.
Methodology: RLSPE and RLSPI
The authors propose a two-stage algorithmic approach to robust learning:
- Robust Least Squares Policy Evaluation (RLSPE($\lambda$)): This is a multi-step, online algorithm designed to estimate the robust value of a specific policy. The key innovation is the definition of an approximate robust TD($\lambda$) operator. Because robust operators are non-linear and difficult to estimate from samples, this approximation makes robust evaluation computationally tractable for online learning.
- Robust Least Squares Policy Iteration (RLSPI): This iterative algorithm uses RLSPE to evaluate a policy and then improves it by acting greedily with respect to the estimated robust values.
Theoretical Insights
The primary challenge in robust RL is the non-linearity introduced by the "infimum" (the search for the worst-case scenario). In standard RL, the Bellman operator is linear, allowing for straightforward application of least squares methods. The authors overcome this by proving that their approximate robust operator is a contraction mapping in a weighted Euclidean norm. This theoretical foundation allows them to provide formal error bounds, showing that the learned policy's distance from optimality is bounded by a constant related to the feature set's representational power.
Experimental Evaluation
The RLSPI algorithm was evaluated on several OpenAI Gym benchmarks, including CartPole, MountainCar, and FrozenLake.
As shown in the experimental results, RLSPI significantly outperforms non-robust baselines like standard LSPI and Q-learning when environment conditions deviate from the training setup. Specifically, RLSPI demonstrates superior resilience to both stochastic action noise and systematic parameter shifts.
Notably, RLSPI performs comparably to "Soft-Robust" deep RL methods (which use neural networks) despite its simpler linear architecture. This suggests that linear function approximation can be a "sweet spot" for robust RL, offering a balance between representational power and analytical tractability.
Limitations and Future Directions
While RLSPI provides strong guarantees for linear function approximation, many vision-based tasks require the non-linear power of deep neural networks. Additionally, the algorithm currently requires a pre-defined "uncertainty radius" ($r$) that must be manually tuned. Future work will focus on extending these theoretical results to non-linear architectures, investigating sample complexity, and exploring offline robust RL settings where agents learn from fixed datasets rather than live simulators.
Summary updated