User-Oriented Robust Reinforcement Learning

Haoyi You, Beichen Yu, Haiming Jin, Zhaoxing Yang, Jiahui Sun

Abstract

Standard robust Reinforcement Learning (RL) often produces overly conservative policies by optimizing for the absolute worst-case scenario. This paper introduces User-Oriented Robust RL (UOR-RL), a framework that allows users to balance performance across different environments based on their specific preferences and the actual probability of those environments occurring. By generalizing average-case and worst-case metrics, UOR-RL provides a tunable spectrum of robustness that avoids the pitfalls of extreme conservatism.

Introduction

In the transition from simulation to the real world (sim-to-real), RL agents frequently encounter environments that differ from their training conditions. Traditional approaches utilize max-min robustness, which optimizes for the single worst possible environment. However, this method suffers from two major flaws:

  1. Over-Conservatism: Optimizing solely for rare, extreme scenarios (e.g., total power failure) can significantly degrade performance in common, everyday situations.
  2. Ignoring User Preference: Standard robust RL offers a one-size-fits-all solution, ignoring that different users have varying risk tolerances—some may prefer aggressive performance while others prioritize safety.

The User-Oriented Robustness (UOR) Framework

The authors propose the User-Oriented Robustness (UOR) metric, defined as a weighted sum of performance across various environments. The weights are determined by three key factors:

  • Environment Distribution: The likelihood of an environment occurring.
  • Relative Performance: The policy's performance in a specific environment compared to others.
  • Preference Function ($W$): A user-defined function that dictates the weight given to "bad" environments. By adjusting a single parameter ($k$), users can navigate a spectrum from "Average-Case" ($k=0$) to "Worst-Case" ($k \to \infty$).

Algorithms

To optimize this metric, two algorithms were developed:

  • DB-UOR-RL (Distribution-Based): Applicable when the environment probability distribution is known, using discretization into "blocks" to calculate the UOR metric.
  • DF-UOR-RL (Distribution-Free): A practical variant for unknown distributions that uses trajectory clustering to estimate the metric dynamically.

Experimental Evaluation

The framework was evaluated using the MuJoCo physics engine across six continuous control tasks: Reacher, Hopper, HalfCheetah, Humanoid, Ant, and Walker 2d. Physical parameters such as body size, density, and friction were varied to create a diverse set of test environments.

Key Results and Insights

UOR-RL demonstrates that robustness is a spectrum rather than a binary choice. By weighting environments based on their performance rank, the agent prioritizes scenarios where it struggles most, aligned with the user's risk tolerance.

Performance and Robustness

DB-UOR-RL outperformed state-of-the-art baselines like EPOpt and MRPO across all MuJoCo tasks when evaluated using the UOR metric ($k=1$). Furthermore, UOR-RL remained competitive even on traditional "worst-case" metrics. For instance, in the Ant task, DF-UOR-RL achieved a score of 3265, significantly higher than MRPO's 2187.

Comparison of performance heatmaps across different RL methods. Darker colors indicate higher performance. UOR-RL methods maintain high performance over a wider range of environment parameters compared to baselines.

Visual analysis via heatmaps confirms that UOR-RL policies maintain high performance over a much broader range of physical conditions (e.g., friction vs. density) than baseline methods.

Detailed performance heatmap for the DF-UOR-RL algorithm, showing consistent high performance across the parameter space.

Conclusion and Future Work

While UOR-RL offers a flexible and robust framework, challenges remain. The computational complexity of block division grows with the dimensionality of environment parameters. Additionally, while the $k$ parameter simplifies preference, real-world preferences may require more complex modeling. Future research will focus on learning preference functions from user behavior and validating these policies on physical robotic hardware.

Summary updated

Abstract

Recently, improving the robustness of policies across different environments attracts increasing attention in the reinforcement learning (RL) community. Existing robust RL methods mostly aim to achieve the max-min robustness by optimizing the policy's performance in the worst-case environment. However, in practice, a user that uses an RL policy may have different preferences over its performance across environments. Clearly, the aforementioned max-min robustness is oftentimes too conservative to satisfy user preference. Therefore, in this paper, we integrate user preference into policy learning in robust RL, and propose a novel User-Oriented Robust RL (UOR-RL) framework. Specifically, we define a new User-Oriented Robustness (UOR) metric for RL, which allocates different weights to the environments according to user preference and generalizes the max-min robustness metric. To optimize the UOR metric, we develop two different UOR-RL training algorithms for the scenarios with or without a priori known environment distribution, respectively. Theoretically, we prove that our UOR-RL training algorithms converge to near-optimal policies even with inaccurate or completely no knowledge about the environment distribution. Furthermore, we carry out extensive experimental evaluations in 4 MuJoCo tasks. The experimental results demonstrate that UOR-RL is comparable to the state-of-the-art baselines under the average and worst-case performance metrics, and more importantly establishes new state-of-the-art performance under the UOR metric.
BibTeX
@inproceedings{you2023user,
  author = {Haoyi You and Beichen Yu and Haiming Jin and Zhaoxing Yang and Jiahui Sun},
  title = {{User-Oriented Robust Reinforcement Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i12.26781},
}
Figure