Real-Time Model Predictive Control and System Identification Using Differentiable Simulation

Sirui Chen, Keenon Werling, Albert Wu, C. Karen Liu

TL;DR

This paper introduces a novel framework for robots to continuously adapt their control and internal models in real-time, even in dynamic environments. By leveraging differentiable physics simulation, the system simultaneously identifies system parameters and plans optimal actions, enabling faster and more robust sim-to-real transfer and adaptation.

What Problem Does This Paper Address?

The "sim-to-real gap" is a persistent challenge in robotics: controllers trained in simulation often fail when deployed on physical robots due to discrepancies between the simulated and real-world dynamics. Existing solutions typically focus on either making controllers more robust to parameter variations or improving simulation fidelity. However, these approaches often fall short when the real-world environment changes dynamically after deployment. This paper tackles the problem of enabling robots to continuously improve their models and control strategies in situ, adapting to unforeseen changes in their environment and internal parameters (like object weight) without requiring extensive retraining or offline calibration.

Methods

The core of the proposed solution is a differentiable physics simulation framework that operates in real-time using a dual-threaded architecture:

  • Modeling Thread (System Identification - SysID): This thread continuously observes the robot's state and uses incoming data to optimize the internal physics model parameters (e.g., mass, inertia, stiffness). It leverages the gradients provided by the differentiable physics engine to efficiently update these parameters. A key innovation here is an adaptive online SysID that assesses the confidence of its parameter estimates. This confidence score helps the system distinguish between uninformative observations and genuine environmental changes, allowing it to either update parameters cautiously or trigger a more aggressive re-estimation.
  • Planning Thread (Optimal Control): This thread uses the current, continuously updated physics model to plan a sequence of future actions. It employs a model-predictive control (MPC) approach, specifically using an iterative Linear Quadratic Regulator (iLQR). A crucial aspect for real-time performance is the adaptive scheduling of the optimization window. The system dynamically adjusts the start time of the planning horizon to ensure that new actions are generated faster than they are consumed, keeping the control plan fresh and responsive.

The system also includes an "active SysID" mode. If the confidence score for parameter estimation remains low for an extended period, the planning thread can be tasked with generating trajectories specifically designed to "excite" the system and provide more informative data for parameter identification.

Datasets

The paper does not introduce new datasets. Instead, it evaluates its method on a series of simulated robotic tasks and one real-world robot experiment. The simulated environments are designed to mimic real-world challenges by introducing:

  • Unknown and changing system parameters: Masses, damping coefficients, spring stiffness, and moments of inertia are varied or unknown.
  • Sensing and actuating errors: Gaussian noise is added to observations and actions to simulate real-world imperfections.

The real-world experiment involves a Rokae Xmate3P robotic arm.

Key Insights and Ideas

  • Continuous Adaptation is Key: The most significant insight is that robots don't need to be perfect from the start. Continuous, real-time adaptation of both the internal model and the control policy is a powerful way to achieve robust performance in dynamic environments.
  • Differentiable Physics as a Unifier: Using a differentiable physics engine is central. It allows for efficient gradient-based optimization for both system identification and control planning simultaneously, which is a significant advantage over methods that treat these as separate or sequential problems.
  • Confidence is Crucial for Robust SysID: Simply updating parameters with every new observation can lead to instability, especially with noisy data. The confidence assessment mechanism is vital for making SysID robust and preventing erroneous updates.
  • Active Exploration for Better Models: The "active SysID" mode is a clever way to overcome situations where passive observation isn't yielding enough information. By actively controlling the robot to generate "parametrically exciting" data, the system can accelerate its own learning process.
  • Scheduling is Paramount for Real-Time Control: The adaptive planning window is a practical solution to the challenge of generating control commands faster than they are executed, a common bottleneck in real-time MPC.

Key Results

  • Superior Performance: The proposed method consistently outperformed several baseline methods (Naive, Smooth, Weighted, UP-OSI, DuST) across various simulated tasks (Cartpole, Inverted Double Pendulum, Robot Arm, Elastic Rod) and on a real robot.
  • Faster Task Completion: In several experiments, the proposed method achieved significantly faster task completion times compared to baselines, demonstrating the benefit of accurate and adaptive modeling for control. For example, in the Cartpole task, it was faster and had fewer failures than a fixed-start-time approach.
  • Accurate Parameter Identification: The system accurately identified changing parameters like masses, damping coefficients, and spring stiffness, often outperforming baselines that struggled with these dynamics.
  • Real-World Validation: Experiments on a physical robot arm showed that the method could adapt to payload changes, maintaining precise control where an offline SysID method failed.
  • Efficiency: The use of differentiable physics and adaptive scheduling enabled both SysID and control to run efficiently in real-time. The paper notes their controller was "80x faster than the stochastic optimization based controller implemented in DuST."

Limitations and Future Work

  • Contact-Rich Environments: The authors acknowledge that optimizing complex trajectories involving contact remains challenging. The convergence of the iLQR planner can be slow and the loss landscape can be "rough" in these scenarios. Developing a more effective controller for contact is identified as a critical next step.
  • Hyperparameter Sensitivity: While the method is shown to be relatively robust to some hyperparameters (like confidence thresholds), it can be sensitive to the MPC control horizon (T), which may require hand-tuning.
  • Future Work: The paper suggests extending the framework to handle contact-rich scenarios. Further exploration of adaptive control strategies and more sophisticated methods for parameter identification in complex dynamic systems are also implied.

Summary updated

Abstract

Transferring a controller from a simulated environment to a physical system is regarded as a challenging problem in robotics. We present a method for continuous improvement of modeling and control after deploying the robot to a dynamically-changing target environment. We develop a differentiable physics simulation framework that simultaneously performs online system identification and optimal control using the incoming observations from the target environment in real time. To ensure robust system identification against noisy observations, we devise an algorithm to assess the confidence of our estimated parameters using numerical analysis of the dynamic equations. To ensure real-time optimal control, we adapt start time of the optimization window so that the optimized actions can be replenished ahead of consumption, while staying as up-to-date with new information as possible. The constantly re-planning based on a constantly improving model allows the robot to swiftly adapt to the changing environment using real-world data in a sample-efficient way. Thanks to a fast differentiable physics simulator, both system identification and control can be solved efficiently in real time. We demonstrate our method on a set of examples in simulation and on a real robot. Our method can outperform all baseline methods in different experiments.
BibTeX
@article{chen2022real,
  author = {Sirui Chen and Keenon Werling and Albert Wu and C. Karen Liu},
  title = {{Real-Time Model Predictive Control and System Identification Using Differentiable Simulation}},
  journal = {IEEE Robotics and Automation Letters},
  year = {2022},
  doi = {10.1109/lra.2022.3226027},
}
Figure