Real-time Model Predictive Control and System Identification Using Differentiable Physics Simulation

Sirui Chen, Keenon Werling, C. Karen Liu

TL;DR

This paper introduces a novel framework that allows robots to continuously improve their understanding of their environment and their control strategies in real-time. By combining differentiable physics simulation with a dual-threaded approach for system identification and model predictive control, robots can adapt to dynamic changes and learn more efficiently from real-world data.

What Problem Does This Paper Address?

The "sim-to-real gap" is a persistent challenge in robotics. Controllers developed in simulation often perform poorly when deployed on physical robots due to discrepancies between the simulated and real-world dynamics. While some approaches focus on making controllers more robust or improving simulation fidelity, this paper tackles the problem of continuous adaptation after deployment. Specifically, it addresses how a robot can adapt its internal model of its own dynamics and its control strategy simultaneously and in real-time, especially in environments that are not static but change over time (e.g., a quadruped robot encountering different terrains or a manipulator handling objects of varying weights).

Methods

The core of the proposed solution is a differentiable physics simulation framework that enables simultaneous online system identification (SysID) and optimal control. This is achieved through a dual-threaded architecture:

  • Modeling Thread: This thread continuously optimizes the robot's system parameters (e.g., mass, inertia, stiffness) by minimizing the difference between simulated and observed states. It leverages the gradients provided by the differentiable physics engine to efficiently update these parameters. A key innovation here is an adaptive online SysID method that assesses the confidence of its parameter estimates. This confidence score helps the system decide whether to accept new parameter estimates, discard them, or even switch to an "active SysID" mode to actively generate more informative observations.
  • Planning Thread: This thread performs real-time optimal control using Model Predictive Control (MPC), specifically employing an iterative Linear Quadratic Regulator (iLQR). A critical challenge for real-time control is ensuring that the planned actions are computed faster than they are consumed. The paper addresses this by adaptively scheduling the optimization window, ensuring that new plans are always available and up-to-date.

The integration of these two threads allows the robot to learn from its environment and adapt its control policy concurrently, leading to sample-efficient learning and adaptation.

Datasets

Not discussed in the paper. The experiments are conducted in simulated environments and on a physical robot, with parameters being identified online rather than trained on pre-existing datasets.

Key Insights and Ideas

  • Differentiable Physics as a Unified Tool: The paper highlights the power of differentiable physics engines not just for control but also for simultaneous system identification. This unified approach simplifies the pipeline and allows for efficient gradient-based optimization for both tasks.
  • Confidence-Aware System Identification: Simply performing online SysID can lead to unstable parameter estimates, especially with noisy or uninformative observations. The proposed confidence score mechanism is crucial for robust adaptation, allowing the system to intelligently decide when and how to update its model.
  • Active System Identification: When passive observations are insufficient, the system can proactively influence its own behavior to generate "parametrically exciting" data. This is a powerful concept for accelerating learning in challenging scenarios.
  • Adaptive Scheduling for Real-Time Control: The challenge of keeping MPC plans fresh in real-time is addressed by dynamically adjusting the planning horizon. This ensures that the robot always has a relevant set of actions to execute.
  • Sample Efficiency: By continuously learning and adapting, the robot can achieve its goals with significantly fewer real-world interactions compared to methods that rely on offline training or less adaptive online approaches.

Key Results

  • Improved Control Performance: In all tested scenarios (cartpole, inverted double pendulum, robot arm, elastic rod), the proposed method significantly outperformed baseline methods in terms of task completion time and accuracy. For instance, in the cartpole experiment, the proposed method achieved a task completion time of 6.31 seconds, compared to baselines that took longer or even failed.
  • Accurate Parameter Estimation: The system demonstrated accurate and rapid identification of various physical parameters, including masses, damping coefficients, moments of inertia, and spring stiffness, even when these parameters changed dynamically during the experiment.
  • Robustness to Noise: The confidence score mechanism helped the system maintain stable parameter estimates and effective control despite simulated sensor and actuator noise.
  • Real-World Validation: Experiments on a physical Rokae Xmate3P robotic arm demonstrated the framework's effectiveness in adapting to changes in end-effector payload mass, maintaining precise control with minimal tracking error.
  • Speed: The differentiable physics engine allowed for efficient real-time computation of both system identification and control, making the approach practical for dynamic robotic applications.

Limitations and Future Work

  • Contact-Rich Environments: The paper acknowledges that optimizing complex trajectories involving contact remains challenging. The convergence of the iLQR planner can be slow and the quality of solutions can be inconsistent due to ambiguous contact gradients and rough loss landscapes. Developing a more effective controller for contact scenarios is identified as a critical next step.
  • Hyperparameter Sensitivity: While the system showed robustness to some hyperparameters, it was noted to be more sensitive to the MPC control horizon (T), which may require hand-tuning for specific applications.
  • Future Work: The authors suggest extending the framework to handle contact-rich scenarios and exploring more advanced control strategies for improved performance in such environments.

Summary updated

Abstract

Developing robot controllers in a simulated environment is advantageous but transferring the controllers to the target environment presents challenges, often referred to as the "sim-to-real gap". We present a method for continuous improvement of modeling and control after deploying the robot to a dynamically-changing target environment. We develop a differentiable physics simulation framework that performs online system identification and optimal control simultaneously, using the incoming observations from the target environment in real time. To ensure robust system identification against noisy observations, we devise an algorithm to assess the confidence of our estimated parameters, using numerical analysis of the dynamic equations. To ensure real-time optimal control, we adaptively schedule the optimization window in the future so that the optimized actions can be replenished faster than they are consumed, while staying as up-to-date with new sensor information as possible. The constant re-planning based on a constantly improved model allows the robot to swiftly adapt to the changing environment and utilize real-world data in the most sample-efficient way. Thanks to a fast differentiable physics simulator, the optimization for both system identification and control can be solved efficiently for robots operating in real time. We demonstrate our method on a set of examples in simulation and show that our results are favorable compared to baseline methods.
BibTeX
@article{chen2022real,
  author = {Sirui Chen and Keenon Werling and C. Karen Liu},
  title = {{Real-time Model Predictive Control and System Identification Using Differentiable Physics Simulation}},
  journal = {IEEE Robotics and Automation Letters},
  year = {2022},
  doi = {10.1109/lra.2022.3226027},
}
Figure