Robust Reinforcement Learning via Progressive Task Sequence

Yike Li, Yunzhe Tian, Endong Tong, Wenjia Niu, Jiqiang LiuIJCAI 2023 · DOI

TL;DR

This paper tackles the challenge of making reinforcement learning (RL) agents robust to real-world uncertainties by reformulating the problem from a "worst-case" optimization to a "max-expectation" approach. They introduce DRRL, a novel framework that uses dynamic, progressive multi-task learning to train agents that are not only resilient to worst-case scenarios but also perform well across a broader range of conditions, leading to more stable training and better generalization.

What Problem Does This Paper Address?

Reinforcement learning (RL) agents often achieve impressive performance in simulated environments. However, a significant "reality gap" exists between simulations and the real world, meaning policies trained in simulation can fail when deployed in real-world scenarios due to subtle differences in dynamics, environmental shifts, or sensor noise. This is particularly critical in safety-sensitive applications like autonomous driving or medical robotics.

Traditional approaches to robust RL often frame the problem as a "max-min" optimization: maximizing the agent's reward under the worst possible perturbations. While this aims for robustness, it can lead to overly conservative policies that sacrifice performance in non-worst-case scenarios and can also result in unstable training processes. The authors argue that this conservative approach stems from the agent's limited "knowledge space," which is confined only to the worst-case perturbations.

Methods

The paper proposes two main innovations:

  1. Max-Expectation Formulation: Instead of the max-min approach, they redefine robust RL as a "max-expectation" optimization. The goal is to find a policy that maximizes the expected reward across all feasible perturbations, not just the worst ones. This expands the agent's "knowledge space," encouraging it to learn policies that are more generally capable and less prone to extreme conservatism.
  2. Dynamic Robust Reinforcement Learning (DRRL): To solve this new formulation, they introduce DRRL, a framework that treats robust RL as a progressive multi-task learning problem.
  3. Task Generation: A genetic algorithm (GA) is used to dynamically generate a population of candidate "tasks" (i.e., perturbed environments). The fitness of each task is determined by how much the current policy improves its reward on that task.
  4. Task Evaluation & Feasibility: A crucial step is filtering these generated tasks to select "feasible tasks." These are tasks that offer a reasonable expected reward (not too low, not too high) for the current policy, ensuring they are challenging enough to promote learning but not so difficult as to destabilize training. This is defined by a "Wasserstein constraint" to keep perturbations within a reasonable bound around the original simulation dynamics.
  5. Progressive Sequencing: The difficulty of these feasible tasks is progressively increased over training iterations. This "easy-to-hard" curriculum helps the agent build a robust foundation and gradually adapt to more complex perturbations, leading to improved training stability and generalization.
  6. Robust Policy Network: They also propose an improved policy network that takes not only the current state but also the previous state and action as input. This allows the agent to infer latent environmental parameters (like friction) indirectly, leading to better generalization.

Datasets

The paper evaluates DRRL on several standard benchmarks:

  • MuJoCo Environments: Hopper, HalfCheetah, and Walker2d. These are high-dimensional continuous control tasks.
  • CarRacing Game: A new benchmark used for evaluating the robust policy network, involving perturbations to friction, gas, steering, and braking.

Key Insights and Ideas

  • Shifting from "Worst-Case" to "Average-Case" Robustness: The core insight is that optimizing for the absolute worst-case can be detrimental. By aiming for a high expected reward across a range of perturbations, agents can achieve better overall robustness and avoid overly conservative behaviors.
  • Curriculum Learning for Robustness: The progressive task sequencing is key. Starting with easier perturbations and gradually increasing difficulty allows for more stable learning and prevents the agent from being overwhelmed by extreme scenarios too early.
  • Dynamic Task Generation is Crucial: Simply applying random perturbations isn't enough. The GA-based generation and evaluation mechanism ensures that the tasks are relevant to the current policy's capabilities, making the learning process more efficient and effective.
  • History Matters for Robustness: The improved policy network, which considers past states and actions, allows the agent to implicitly learn about environmental dynamics, leading to better adaptation.

Key Results

  • Improved Training Stability: DRRL consistently shows more stable reward increases and smaller standard deviations in rewards across training iterations compared to baselines.
  • Superior Robustness: DRRL outperforms state-of-the-art methods (TRPO, RARL, RAP-3, DR) in various test conditions, including adversarial perturbations and changes in mass and friction. For instance, it achieved significant improvements (e.g., 12.04% in Walker2d under no perturbations, 66.39% in Hopper under adversarial perturbations).
  • Effective Policy Network: An ablation study shows that the proposed robust policy network is vital for DRRL's performance, outperforming a variant using a traditional policy network.
  • Generalization: The method demonstrates strong generalization, particularly in the Walker2d environment, where it performs well across a wide range of friction and mass variations.

Limitations and Future Work

  • Hyperparameter Tuning: The paper mentions that hyper-parameters like $W_{min}$ and $W_{max}$ for defining feasible tasks need careful tuning.
  • Computational Cost: The GA-based task generation and evaluation process can be computationally intensive.
  • Future Work: The authors suggest that their framework can be extended to more complex scenarios and that further research into more efficient task generation mechanisms could be beneficial. The proposed robust policy network could also be explored in other RL domains.

Summary updated

BibTeX
@inproceedings{li2023robust,
  author = {Yike Li and Yunzhe Tian and Endong Tong and Wenjia Niu and Jiqiang Liu},
  title = {{Robust Reinforcement Learning via Progressive Task Sequence}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/51},
}
Figure