Faster Model Predictive Control via Self-Supervised Initialization Learning

Zhaoxin Li, Letian Chen, Rohan R. Paleja, Subramanya Nageshrao, Matthew C. Gombolay

Abstract

Optimization for robot control tasks, spanning various methodologies, includes Model Predictive Control (MPC). However, the complexity of the system, such as non-convex and non-differentiable cost functions and prolonged planning horizons often drastically increases the computation time, limiting MPC's real-world applicability. Prior works in speeding up the optimization have limitations on solving convex problem and generalizing to hold out domains. To overcome this challenge, we develop a novel framework aiming at expediting optimization processes. In our framework, we combine offline self-supervised learning and online fine-tuning through reinforcement learning to improve the control performance and reduce optimization time. We demonstrate the effectiveness of our method on a novel, challenging Formula-1-track driving task, achieving 3.9% higher performance in optimization time and 3.6% higher performance in tracking accuracy on challenging holdout tracks.

Introduction

Model Predictive Control (MPC) is a cornerstone of modern robotics, valued for its ability to optimize control actions over a future horizon while explicitly handling system constraints. Despite its robustness, MPC is computationally intensive. When faced with non-smooth dynamics or intricate obstacle environments, researchers often employ gradient-free or sampling-based solvers. While these solvers are resilient, they are frequently too slow for real-time applications, particularly when sudden state changes—such as sharp turns—demand rapid re-optimization.

Traditional "warm-start" strategies typically initialize the solver with the solution from the previous timestep. However, this heuristic often fails during high-dynamic maneuvers. While pure machine learning approaches like Behavior Cloning (BC) can predict control sequences, they suffer from covariance shift and do not inherently optimize for the solver's convergence speed.

Methodology

To address these challenges, we propose a two-phase framework to train a specialized warm-start policy ($\pi_\theta^{warm}$). This policy provides high-quality initial guesses to the MPC solver, bridging the gap between fast neural network inference and rigorous classical control.

The proposed framework consists of offline behavior cloning, online fine-tuning with RL and DAgger, and final zero-shot testing.

Phase 1: Offline Behavior Cloning

The framework begins by collecting "expert" demonstrations from a high-iteration, slow MPC solver. A neural network is trained via self-supervised learning to mimic these expert control sequences based on the current state. This phase establishes a baseline policy that understands the basic task structure but remains limited by the expert's own suboptimality.

Phase 2: Online Fine-Tuning

The pre-trained policy is then moved into an online environment for fine-tuning using Reinforcement Learning (RL). A critical innovation here is the reward function, which explicitly penalizes both tracking error and MPC optimization time. To maintain stability and prevent the policy from drifting into infeasible regions (the covariance shift problem), we integrate DAgger (Dataset Aggregation). This allows the system to periodically query the expert MPC for corrective labels during online exploration.

Experimental Evaluation

We evaluated the framework across two challenging domains:

  1. Formula 1 Path-Tracking: Utilizing a deterministic COBYLA solver on complex tracks like the Nürburgring and Melbourne (scaled 10:1).
  2. Obstacle-Rich Navigation: Utilizing a sampling-based MPPI solver in environments with randomly generated polygonal obstacles.
Example of a complex racing track (Zandvoort) used for evaluating zero-shot generalization.

Key Results

Our method significantly outperformed standard initialization baselines, such as "All-Zero" or "Previous Solution" heuristics:

  • Efficiency: In F1 racing tasks, the framework achieved up to a 21.6% reduction in optimization time.
  • Accuracy: Tracking accuracy on unseen tracks improved by 34.1%. Traditional solvers often timed out or became trapped in local minima on complex segments where our warm-started solver succeeded.
  • Safety and Smoothness: In navigation tasks, the method achieved zero collisions on test maps (a 100% improvement over baselines), while also improving path efficiency by 12.8% and steering smoothness by 7.2%.

Conclusion

By treating optimization speed as a learnable reward, this framework successfully accelerates MPC solvers without sacrificing their inherent safety and constraint-handling capabilities. Future work will focus on extending this approach to higher-degree-of-freedom systems, such as aerial drones and robotic manipulators, and testing under realistic, noisy sensing conditions.

Summary updated

Abstract

Optimization for robot control tasks, spanning various methodologies, includes Model Predictive Control (MPC). However, the complexity of the system, such as non-convex and non-differentiable cost functions and prolonged planning horizons often drastically increases the computation time, limiting MPC's real-world applicability. Prior works in speeding up the optimization have limitations on solving convex problem and generalizing to hold out domains. To overcome this challenge, we develop a novel framework aiming at expediting optimization processes. In our framework, we combine offline self-supervised learning and online fine-tuning through reinforcement learning to improve the control performance and reduce optimization time. We demonstrate the effectiveness of our method on a novel, challenging Formula-1-track driving task, achieving 3.9\% higher performance in optimization time and 3.6\% higher performance in tracking accuracy on challenging holdout tracks.
BibTeX
@misc{li2024faster,
  author = {Zhaoxin Li and Letian Chen and Rohan R. Paleja and Subramanya Nageshrao and Matthew C. Gombolay},
  title = {{Faster Model Predictive Control via Self-Supervised Initialization Learning}},
  year = {2024},
  eprint = {2408.03394},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.03394},
}
Figure