Performance Guaranteed MPC Policy Approximation via Cost Guided Learning

Chenchen Zhou, Yi Cao, Shuang-Hua YangIEEE Control. Syst. Lett 2024 · DOI

TL;DR

This paper introduces a novel "cost-guided learning" approach to approximate Model Predictive Control (MPC) policies using neural networks. Instead of minimizing fitting errors, this method directly minimizes the operational cost incurred by the approximate policy, leading to tighter performance guarantees and demonstrably better closed-loop performance on a benchmark system.

What Problem Does This Paper Address?

Model Predictive Control (MPC) is a powerful technique for optimizing dynamic system performance, but its computational demands can be prohibitive for real-time implementation, especially for nonlinear systems. A common workaround is to pre-compute an optimal MPC policy and approximate it using function approximators like neural networks. However, existing methods typically train these approximators by minimizing the difference between their output and the true MPC actions on training data (error-guided learning). This approach has a critical flaw: minimizing fitting error doesn't directly translate to minimizing actual operational cost. Small fitting errors can lead to significant performance degradation or even constraint violations in practice, particularly for "sharp" or constrained optima. This paper addresses this gap by proposing a method that directly optimizes for operational cost.

Methods

The core innovation is the cost-guided learning approach. This method leverages the Lagrangian function of the MPC optimization problem. The Lagrangian captures both the cost and constraint satisfaction aspects of the problem. The authors propose a loss function, $L_w$, that is derived from the Hessian of the Lagrangian. This loss function directly penalizes deviations from the optimal control action that lead to higher operational costs, effectively focusing the learning process on critical regions of the state space where errors are most costly.

The paper also introduces a control policy evaluation indicator based on the discrepancy of Lagrangian values, which is shown to provide a tighter upper bound on optimality loss compared to traditional error-guided methods. Theoretical analysis is provided to support these claims, demonstrating that cost-guided learning offers better guarantees on the performance of the approximated policy.

Datasets

The paper uses a Continuous Stirred-Tank Reactor (CSTR) benchmark problem for evaluation. The model is described, and the stage cost is defined. For training, they generate state-action data pairs using a grid-based sampling approach. The MPC problem is solved for each sample to obtain the ground truth optimal actions. No new benchmark datasets are introduced; rather, a standard benchmark is used to demonstrate the effectiveness of the proposed method.

Key Insights and Ideas

  • Directly optimizing for cost is superior to imitating expert actions: The fundamental insight is that the true objective of control is to minimize operational cost, not to perfectly mimic an expert's actions. Cost-guided learning aligns the training objective with the ultimate goal.
  • Lagrangian Hessian is key to cost sensitivity: The Hessian of the Lagrangian provides crucial information about how sensitive the optimal solution is to changes in control inputs, particularly in the presence of constraints. This sensitivity is leveraged to weight errors appropriately during training.
  • Tighter performance guarantees: Theoretical analysis shows that cost-guided learning provides a tighter upper bound on the optimality loss compared to traditional error-guided learning, meaning the approximated policy is more likely to perform close to the true MPC policy.
  • Focus on high-cost regions: The cost-guided approach implicitly prioritizes learning accurate control actions in regions of the state space where deviations from the optimum are most costly, leading to more robust performance.

Key Results

The paper demonstrates the effectiveness of the cost-guided learning approach through experiments on the CSTR benchmark.

  • Improved Closed-Loop Performance: The approximate policies trained with cost-guided learning ($L_w$) and Lagrangian-based loss ($L_{lag}$) achieved significantly better closed-loop performance (lower $J - J^*$) compared to the policy trained with error-guided learning ($L_{mse}$).
  • Lagrangian Loss Outperforms: The policy trained with the Lagrangian loss ($L_{lag}$) showed the best average performance, followed closely by the cost-guided loss ($L_w$).
  • Training Time: Error-guided and cost-guided learning had similar offline training times, while Lagrangian loss required longer training.
  • Inference Time: All approximated policies offered substantial speedups in online inference time compared to the full MPC, making them suitable for real-time applications.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Data Requirements: The cost-guided learning approach requires a significant amount of data to train the neural network approximator effectively. This can be a bottleneck for applications where data collection is difficult or expensive.
  • Approximating First Step vs. Closed-Loop: The current work focuses on approximating the first step of the MPC plan. Approximating the full closed-loop MPC policy could improve robustness but presents additional challenges.
  • Real-World System Challenges: Real-world systems often involve noise, uncertainties, and sensor errors, which can degrade policy performance. Future work should explore combining the proposed method with techniques like stochastic MPC to handle these issues.
  • Generalizability: The paper suggests that the core idea of connecting fitting errors with control objectives could be broadly applied to other performance-driven learning-based control paradigms, such as offline reinforcement learning and inverse reinforcement learning.

Summary updated

BibTeX
@article{zhou2024performance,
  author = {Chenchen Zhou and Yi Cao and Shuang-Hua Yang},
  title = {{Performance Guaranteed MPC Policy Approximation via Cost Guided Learning}},
  journal = {IEEE Control. Syst. Lett.},
  year = {2024},
  doi = {10.1109/lcsys.2024.3379115},
}
Figure