Robust Zero-Sum Deep Reinforcement Learning

Olalekan P. Ogunmolu, Nicholas R. Gans, Tyler H. SummersarXiv 2017 · arXiv · OpenReview

TL;DR

This paper introduces a novel "minimax iterative dynamic game" (iDG) framework to design robot control policies that are robust to disturbances and model uncertainties. By framing control as a two-player zero-sum game, they quantify policy brittleness and develop an algorithm that converges to a saddle-point equilibrium, leading to more reliable performance in real-world, high-risk scenarios.

What Problem Does This Paper Address?

Modern control policies, especially those learned through deep reinforcement learning (DRL), are powerful but often brittle. They can perform exceptionally well under ideal conditions but degrade significantly when faced with unexpected disturbances, slight variations in the environment (model mismatch), or uncertainties in the system's dynamics. This brittleness makes them unsuitable for high-risk applications where failure can have severe consequences. The paper aims to address this by developing methods to both quantify a policy's susceptibility to these issues and to actively design policies that are inherently robust.

Methods

The core contribution is the Minimax Iterative Dynamic Game (iDG) framework. This approach reframes the control problem as a two-player, zero-sum game:

  1. Policy Sensitivity Quantification: Before designing a robust policy, the authors propose a method to measure how sensitive a given policy is to adversarial disturbances. This involves pitting a "disturbing agent" against the nominal policy and observing the performance degradation. The magnitude of the disturbance that causes significant degradation provides a measure of the policy's robustness.
  2. Minimax Iterative Dynamic Game (iDG): The iDG framework models the interaction between a nominal agent (the robot controller) and an adversarial agent (representing disturbances/uncertainties) as a finite-horizon, two-player zero-sum Markov game.
  3. Alternating Best Response: Each agent takes turns executing an "opposite reaction" to the other. This creates a concave-convex problem, and the goal is to find a saddle-point equilibrium.
  4. Iterative Optimization: The framework iteratively updates both the nominal control policy and the adversarial policy. This process is inspired by iterative LQG (iLQG) and differential dynamic programming (DDP) but adapted for the two-player game.
  5. Local Control Laws: The iDG generates local control laws that ensure convergence to a saddle-point equilibrium, effectively creating a robust policy.
  6. Regularization: To handle potential issues with Hessian invertibility in nonlinear systems, they introduce an improved regularization technique and a schedule for adjusting the regularization parameter.

Datasets

The paper does not explicitly mention using pre-defined datasets for training. Instead, it focuses on:

  • Simulation Environment: Experiments are conducted using a KUKA youbot robot simulated in Gazebo.
  • Task-Specific Data Generation: For the peg insertion task, data is generated by running the PR2 robot (simulated in MuJoCo) with learned policies and adversarial controllers.
  • Robot Dynamics: The complete kinematic and dynamic model of the KUKA youbot is used.

Key Insights and Ideas

  • Quantifying Brittleness is Key: Understanding how brittle a policy is before trying to fix it is crucial. The proposed sensitivity quantification provides a concrete metric (the $\gamma^*$ value).
  • Adversarial Training as a Robustness Tool: Framing the problem as a zero-sum game where an adversary actively tries to disrupt the system is a powerful way to force the controller to learn robust strategies.
  • Saddle Point Equilibrium for Robustness: The concept of a saddle point in a dynamic game directly translates to a state where neither the controller nor the adversary can unilaterally improve their outcome, leading to a stable, robust equilibrium.
  • iDG as a Meta-Algorithm: The iDG framework is presented as a general approach that can be applied to various policy learning methods (model-free RL, model-based RL, iLQG, DDP), making it broadly applicable.
  • Trade-off Between Speed and Robustness: The iDG approach, while more robust, takes slightly longer to converge (5-7 iterations) compared to standard iLQG (3-4 iterations), highlighting a practical trade-off.

Key Results

  • Peg Insertion Task: For the PR2 robot peg insertion task, the iDG-trained policy showed significantly improved robustness. Adversarial inputs that caused sharp degradation (instability) in standard policies had a much smaller effect on the iDG-trained policy, with performance degradation only occurring at very low $\gamma$ values.
  • KUKA Youbot Navigation: On the KUKA youbot, the iDG formulation resulted in smoother trajectories and less arm vibration compared to standard iLQG, even when the arm dynamics were not fully modeled. The iDG policies successfully reached the goal state, demonstrating robustness to unmodeled dynamics.
  • Robustness Parameter ($\gamma$): The parameter $\gamma$ effectively controls the strength of the adversarial disturbance. The paper shows that for values of $\gamma$ above a certain threshold (e.g., $\gamma > 1.5$ for the PR2, $\gamma \in [1.5, 10]$ for the Youbot), the iDG policies maintain good performance, while below this threshold, performance degrades, indicating the critical robustness value.

Limitations and Future Work

  • Computational Cost: The iDG approach requires more iterations than standard iLQG, indicating a higher computational cost for achieving robustness.
  • Local Optima: The paper acknowledges that the iDG algorithm might converge to a locally optimal saddle point, rather than a globally optimal one, especially in complex, continuous state-action spaces.
  • Approximation of Dynamics: While the iDG framework is designed to handle uncertainties, the performance still relies on the accuracy of the underlying system dynamics model used for optimization.
  • Future Work: The authors suggest that the iDG scenario is extensible to other systems and cost functions. They also hint at further research into the theoretical underpinnings and applications of their minimax framework.

Summary updated

Abstract

Multistage decision policies provide useful control strategies in high-dimensional state spaces, particularly in complex control tasks. However, they exhibit weak performance guarantees in the presence of disturbance, model mismatch, or model uncertainties. This brittleness limits their use in high-risk scenarios. We present how to quantify the sensitivity of such policies in order to inform of their robustness capacity. We also propose a minimax iterative dynamic game framework for designing robust policies in the presence of disturbance/uncertainties. We test the quantification hypothesis on a carefully designed deep neural network policy; we then pose a minimax iterative dynamic game (iDG) framework for improving policy robustness in the presence of adversarial disturbances. We evaluate our iDG framework on a mecanum-wheeled robot, whose goal is to find a ocally robust optimal multistage policy that achieve a given goal-reaching task. The algorithm is simple and adaptable for designing meta-learning/deep policies that are robust against disturbances, model mismatch, or model uncertainties, up to a disturbance bound. Videos of the results are on the author's website, http://ecs.utdallas.edu/~opo140030/iros18/iros2018.html, while the codes for reproducing our experiments are on github, https://github.com/lakehanne/youbot/tree/rilqg. A self-contained environment for reproducing our results is on docker, https://hub.docker.com/r/lakehanne/youbotbuntu14/
BibTeX
@misc{ogunmolu2017robust,
  author = {Olalekan P. Ogunmolu and Nicholas R. Gans and Tyler H. Summers},
  title = {{Robust Zero-Sum Deep Reinforcement Learning}},
  year = {2017},
  eprint = {1710.00491},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1710.00491},
}
Figure