Safe Imitation Learning of Nonlinear Model Predictive Control for Flexible Robots

Shamil Mamedov, Rudolf Reiter, Seyed Mahdi B. Azad, Ruan Viljoen, Joschka Boedecker, Moritz Diehl, Jan SweversIROS 2024 · OpenReview

TL;DR

This paper introduces a novel framework that combines imitation learning with a safety filter to create a fast and safe controller for flexible robots. By approximating complex Nonlinear Model Predictive Control (NMPC) using a neural network and then ensuring safety with a predictive filter, the approach achieves an eightfold speedup in computation time while maintaining high performance and guaranteeing constraint satisfaction.

What Problem Does This Paper Address?

Flexible robots offer significant advantages for industrial applications, such as safer human-robot collaboration and improved payload-to-mass ratios. However, their inherent flexibility leads to complex, high-dimensional dynamics characterized by oscillations and deflections, making them difficult to control. While Nonlinear Model Predictive Control (NMPC) is a powerful tool for handling these complexities and ensuring safety, its computational demands are often too high for real-time applications. Existing methods to speed up NMPC, like linearization, can lead to overly conservative control. This paper aims to bridge this gap by developing a computationally efficient yet safe controller that approximates NMPC for flexible robots.

Methods

The proposed framework has three core components:

  1. NMPC Formulation and Data Generation: The authors first formulate an NMPC problem for output regulation of a flexible robot, considering its nonlinear dynamics and constraints. This "expert" NMPC is then used in simulation to generate a dataset of expert demonstrations (state-action pairs).
  2. Imitation Learning (IL) for NMPC Approximation: To approximate the NMPC policy, the paper employs the DAgger algorithm. DAgger is chosen over simpler methods like Behavioral Cloning (BC) because it iteratively collects more data during interaction with the environment and the expert, helping to mitigate approximation errors and improve robustness. A neural network is trained to mimic the NMPC's control actions.
  3. Predictive Safety Filter: A key contribution is the integration of a safety filter. This filter acts as a crucial layer between the learned policy and the robot. It takes the action proposed by the neural network and verifies if it can lead to a safe state within a short prediction horizon using a simplified model. If the action is deemed unsafe, the filter modifies it minimally to ensure constraint satisfaction (e.g., avoiding collisions). This filter is designed to be computationally efficient, allowing for real-time operation.

Datasets

The paper uses simulated data generated by the formulated NMPC controller for training the imitation learning model. No new physical robot datasets or benchmarks are introduced; the focus is on simulating a flexible robot arm.

Key Insights and Ideas

  • DAgger's Advantage: The paper highlights DAgger's superiority over other imitation learning methods (like BC) for approximating complex controllers like NMPC. Its iterative data collection strategy is crucial for learning robust policies that can handle the nuances of flexible robot dynamics.
  • Safety Filter as a Crucial Enabler: The safety filter is presented not just as a post-processing step but as an integral part of the framework that makes the learned policy practically usable. It effectively "guarantees" safety, a property that pure imitation learning struggles to provide.
  • Balancing Performance and Computation: The framework demonstrates that it's possible to achieve significant computational speedups (eightfold) without a drastic loss in performance, a critical trade-off for real-time robotics.
  • Model Mismatch Robustness: The proposed approach shows promising robustness to model uncertainties, a common challenge in real-world robotic systems.

Key Results

  • Computational Speedup: The framework achieves an eightfold reduction in control action computation time compared to the original NMPC.
  • Safety Guarantees: The safety filter effectively eliminates constraint violations, making the system significantly safer.
  • Performance: While there's a slight performance trade-off compared to the expert NMPC, the proposed method consistently outperforms state-of-the-art model-free reinforcement learning algorithms like SAC.
  • Robustness: The framework maintains relatively stable performance even when faced with a 10% reduction in the robot's stiffness parameters.

Limitations and Future Work

  • Performance Trade-off: While the framework is safe and fast, there is a slight decrease in performance (e.g., final distance to goal) compared to the ideal NMPC.
  • Data Requirements: The performance of imitation learning is inherently dependent on the quality and quantity of expert data. More data could further improve the learned policy.
  • Future Directions: The authors suggest extending this approach to trajectory tracking tasks and exploring its application to soft robotics, which present even greater control challenges. They also note that further research could investigate more conservative safety filter strategies when combined with RL algorithms like SAC.

Summary updated

Abstract

Flexible robots may overcome some of the industry’s major challenges, such as enabling intrinsically safe human-robot collaboration and achieving a higher payload-to-mass ratio. However, controlling flexible robots is complicated due to their complex dynamics, which include oscillatory behavior and a high-dimensional state space. Nonlinear model predictive control (NMPC) offers an effective means to control such robots, but its significant computational demand often limits its application in real-time scenarios. To enable fast control of flexible robots, we propose a framework for a safe approximation of NMPC using imitation learning and a predictive safety filter. Our framework significantly reduces computation time while incurring a slight loss in performance. Compared to NMPC, our framework shows more than an eightfold improvement in computation time when controlling a three-dimensional flexible robot arm in simulation, all while guaranteeing safety constraints. Notably, our approach out-performs state-of-the-art reinforcement learning methods. The development of fast and safe approximate NMPC holds the potential to accelerate the adoption of flexible robots in industry. The project code is available at: tinyurl.com/anmpc4fr
BibTeX
@inproceedings{mamedov2024safe,
  author = {Shamil Mamedov and Rudolf Reiter and Seyed Mahdi B. Azad and Ruan Viljoen and Joschka Boedecker and Moritz Diehl and Jan Swevers},
  title = {{Safe Imitation Learning of Nonlinear Model Predictive Control for Flexible Robots}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=dIU5dMA3O8},
}
Figure