Minimax Iterative Dynamic Game: Application to Nonlinear Robot Control Tasks

Olalekan P. Ogunmolu, Nicholas R. Gans, Tyler H. SummersIROS 2018 · OpenReview

TL;DR

This paper introduces a novel "minimax iterative dynamic game" (iDG) framework to design robot control policies that are robust to disturbances and model uncertainties. By framing control as a two-player game against an adversary, the iDG approach allows for the quantification of policy brittleness and the development of more reliable control strategies for high-risk applications.

What Problem Does This Paper Address?

Many advanced control policies, especially those used in high-dimensional systems like robots, are brittle. This means they perform well under ideal conditions but can fail catastrophically when faced with disturbances, inaccuracies in their models (model mismatch), or uncertainties about the environment. This brittleness limits their deployment in safety-critical scenarios where reliability is paramount. The paper aims to address this by developing methods to both quantify a policy's susceptibility to these issues and to actively design policies that are inherently more robust.

Methods

The authors propose a two-pronged approach:

  1. Policy Robustness Quantification: They introduce a method to measure how sensitive a given policy is to adversarial disturbances. This involves pitting a "disturbing agent" against the nominal policy, controlled by a parameter $\gamma$. By observing how much performance degrades as $\gamma$ changes, they can determine an upper bound on the policy's robustness. This is framed as a two-player zero-sum game.
  2. Minimax Iterative Dynamic Game (iDG) Framework: To design robust policies, they propose an iterative dynamic game framework. This framework treats the control problem as a continuous, finite-horizon, two-player zero-sum Markov game. The "nominal agent" (the robot controller) tries to minimize a cost function, while the "adversarial agent" tries to maximize it. By iteratively finding a saddle-point equilibrium between these two agents, the iDG framework generates control laws that are robust to disturbances up to a certain bound. This approach is presented as a meta-algorithm adaptable to various existing policy learning methods (e.g., model-free RL, model-based RL, iLQG).

The core of the iDG framework involves solving a series of local, linearized, two-player trajectory optimization problems. They adapt techniques from differential dynamic programming (DDP) and iterative Linear Quadratic Gaussian (iLQG) control to handle the minimax objective. Regularization techniques are also employed to ensure the stability of the optimization process, especially when dealing with non-positive definite Hessians.

Datasets

The paper does not explicitly mention using pre-existing datasets for training. Instead, it focuses on simulation-based experiments using a KUKA youbot robot model. The experiments involve:

  • Simulated Robot Dynamics: A detailed kinematic and dynamic model of a KUKA youbot with mecanum wheels is used, simulated in the Gazebo physics engine.
  • Task-Specific Scenarios: Two main scenarios are evaluated:
  • A peg insertion task (simulated in MuJoCo) for robustness quantification.
  • A goal navigation task for the youbot robot to evaluate the iDG framework's effectiveness in achieving robust control.

Key Insights and Ideas

  • Brittleness as a Quantifiable Metric: The paper frames policy brittleness not as an abstract concept but as a measurable quantity ($\gamma$) that indicates the threshold of adversarial effort before performance degrades unacceptably. This provides a concrete way to compare policies.
  • Adversarial Game for Robustness Design: Treating robustness as a game against an adversary is a powerful paradigm. It allows for the explicit modeling and mitigation of worst-case scenarios, leading to policies that are inherently more resilient.
  • Saddle-Point Equilibrium for Robust Control: The concept of finding a saddle-point equilibrium in a dynamic game directly translates to finding a control policy that is optimal against the worst possible disturbance, thus achieving robustness.
  • iDG as a Meta-Algorithm: The iDG framework is designed to be general, applicable to various existing control and learning algorithms, making it a versatile tool for enhancing robustness.
  • Trade-off Between Speed and Robustness: The results suggest that while iDG might take slightly more iterations than standard iLQG, it offers a significant improvement in robustness, making it a viable alternative when safety and reliability are prioritized over speed.

Key Results

  • Robustness Quantification: Experiments on a peg insertion task showed that for certain disturbance values ($\gamma < 1.5$), the controller's performance degraded sharply, indicating instability. This allowed for the identification of the critical $\gamma$ value where robustness breaks down.
  • iDG for Navigation: When applied to the youbot robot's goal navigation task, the iDG framework produced smoother trajectories and prevented the arm from vibrating compared to standard iLQG, especially for moderate disturbance levels.
  • Performance under Adversarial Input: For $\gamma$ values in the range of 1.5 to 10, the iDG-controlled robot navigated smoothly and reached its goal without significant arm instability. As $\gamma$ decreased below 1.5, the adversarial disturbance became more pronounced, leading to perturbed trajectories and balance issues.
  • Time-Robustness Trade-off: The iDG approach, while achieving better robustness, required more iterations (5-7) compared to standard iLQG (3-4) for the navigation task. This highlights a clear trade-off between optimization speed and achieved robustness.

Limitations and Future Work

  • Local Optimality: The authors acknowledge that the iDG algorithm might converge to a locally optimal saddle-point equilibrium due to approximations in continuous spaces.
  • Computational Cost: The iterative nature of the iDG framework can lead to longer computation times compared to standard optimization methods, as seen in the comparison with iLQG.
  • Model Accuracy: The paper notes that the robot's arm dynamics were not fully included in the model for the navigation task, which still led to some vibration in the iLQG results. Future work could explore incorporating more complete dynamics.
  • Generalizability: While presented as a meta-algorithm, further validation on a wider range of robotic systems and tasks would be beneficial.
  • Extensibility: The authors suggest that the minimax iDG scenario is extensible to other systems that compute gradients of cost or reward functions, implying future work in applying this to more complex scenarios. They also hypothesize that previous work's lack of robustness might stem from issues in value function recursions, suggesting further investigation into these theoretical aspects.

Summary updated

Abstract

Multistage decision policies provide useful control strategies in high-dimensional state spaces, particularly in complex control tasks. However, they exhibit weak performance guarantees in the presence of disturbance, model mismatch, or model uncertainties. This brittleness limits their use in high-risk scenarios. We present how to quantify the sensitivity of such policies in order to inform of their robustness capacity. We also propose a minimax iterative dynamic game framework for designing robust policies in the presence of disturbance/uncertainties. We test the quantification hypothesis on a carefully designed deep neural network policy; we then pose a minimax iterative dynamic game (iDG) framework for improving policy robustness in the presence of adversarial disturbances. We evaluate our iDG framework on a mecanum-wheeled robot, whose goal is to find a ocally robust optimal multistage policy that achieve a given goal-reaching task. The algorithm is simple and adaptable for designing meta-learning/deep policies that are robust against disturbances, model mismatch, or model uncertainties, up to a disturbance bound. Videos of the results are on the author's website: https://goo.gl/JhshTB, while the codes for reproducing our experiments are on github: https://goo.gl/3G2VBy. A self-contained environment for reproducing our results is on docker: https://goo.gl/Bo7MBe.
BibTeX
@inproceedings{ogunmolu2018minimax,
  author = {Olalekan P. Ogunmolu and Nicholas R. Gans and Tyler H. Summers},
  title = {{Minimax Iterative Dynamic Game: Application to Nonlinear Robot Control Tasks}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=eEcrmIx7lh},
}
Figure