Accelerating Model-Free Policy Optimization Using Model-Based Gradient: A Composite Optimization Perspective

Yansong Li, Shuo HanOpenReview 2024 · OpenReview

TL;DR

This paper introduces a novel algorithm that blends model-based and model-free approaches to accelerate optimal control policy learning. By reformulating the problem as a composite optimization task, their method leverages an approximate system model to guide learning, only resorting to expensive model-free updates when necessary, leading to significant reductions in computation.

What Problem Does This Paper Address?

Learning-based control algorithms are powerful but face a fundamental trade-off: model-free methods are data-hungry and can be impractical for physical systems, while model-based methods struggle with complex dynamics and model inaccuracies (model bias). This paper tackles the challenge of efficiently learning optimal control policies for systems with a known linear dynamics component and a small, unknown nonlinear perturbation. The core issue is how to harness the efficiency of model-based approaches without being crippled by model errors, and how to do so with fewer data samples than purely model-free methods.

Methods

The authors propose a "composite optimization" framework. They decompose the total cost function into two parts: a "model-based" component derived from the approximate linear model, and a "model-free" component representing the unknown modeling error. This allows them to formulate the optimization problem as minimizing the sum of a differentiable function (from the model) and a black-box function (the residual error).

Their key algorithmic innovation is a hybrid gradient descent approach. It primarily uses a "compensated model gradient" derived from the approximate linear model. This compensated gradient is intermittently updated with information from a model-free algorithm (specifically, a policy gradient method). When the approximate model's gradient is close to the true gradient, the algorithm stays in a "model-based regime," making rapid progress. When the model error becomes significant, or when backtracking line search fails, the algorithm switches to a "model-free regime" to obtain a more accurate gradient estimate, which is then used to update the compensation term for future model-based steps. This strategy aims to balance the speed of model-based updates with the accuracy of model-free ones.

Datasets

Not discussed in the paper. The experiments use synthetic quadratic and modified LQR problems.

Key Insights and Ideas

  • Composite Optimization as a Unifying Framework: Viewing the optimal control problem as a composite function (model-based + model-free residual) provides a powerful lens for designing hybrid algorithms.
  • Gradient Compensation for Model Bias: Instead of directly correcting the model, the algorithm compensates for model bias in the gradient estimate. This allows it to leverage the structure of the approximate model while still accounting for errors.
  • Adaptive Regime Switching: The algorithm intelligently switches between a fast, model-based regime and a more robust, model-free regime based on the success of backtracking line search and the estimated error. This dynamic adaptation is crucial for efficiency.
  • Backtracking Line Search as a Signal: The failure of standard backtracking line search is used as a critical signal to indicate when the model-based gradient is no longer reliable, triggering a switch to the model-free regime.
  • Theoretical Guarantees for Progress: The paper provides theoretical analysis showing that under certain conditions, the algorithm guarantees sufficient decrease in the objective function, even when operating in the model-based regime.

Key Results

  • Reduced Function Evaluations: Both theoretical analysis and numerical experiments demonstrate that the proposed algorithm requires significantly fewer function evaluations compared to traditional model-free policy gradient methods to reach the same level of suboptimality.
  • Quadratic Functions: For quadratic problems, the algorithm shows substantial speedups, especially when the problem dimension is large.
  • Modified LQR: In a modified LQR setting, the algorithm also outperforms the model-free baseline, though it spends a considerable portion of its time in the model-free regime due to the nature of the problem and parameter choices.
  • Parameter Sensitivity: The performance is shown to be sensitive to parameters like the minimum step size in backtracking line search, highlighting the need for careful tuning to balance staying in the model-based regime and the cost of line search iterations.

Limitations and Future Work

  • "Small" Nonlinear Perturbation: The theoretical analysis and algorithm design assume the nonlinear perturbation is "small." The paper acknowledges that the precise meaning of "small" is important and is further explored in Section 4.3, where it's related to the smoothness constant of the residual cost. The performance with larger perturbations is not explicitly discussed.
  • Hyperparameter Tuning: The effectiveness of the algorithm, particularly the balance between model-based and model-free regimes, depends on hyperparameter choices (e.g., γ, η_min, η_max). The paper explores some of these sensitivities but doesn't offer a fully automated tuning strategy.
  • General Nonlinear Systems: The current formulation and experiments focus on systems with a linear state-space model plus a small nonlinear perturbation and quadratic costs. Extending the framework to more general nonlinear systems and cost functions would be a natural next step.
  • Model Uncertainty Quantification: While the algorithm implicitly handles model uncertainty through compensation, explicit quantification of model uncertainty and its impact on the learning process could be a valuable area for future research.

Summary updated

Abstract

We develop an algorithm that combines model-based and model-free methods for solving a nonlinear optimal control problem with a quadratic cost in which the system model is given by a linear state-space model with a small additive nonlinear perturbation. We decompose the cost into a sum of two functions, one having an explicit form obtained from the approximate linear model, the other being a black-box model representing the unknown modeling error. The decomposition allows us to formulate the problem as a composite optimization problem. To solve the optimization problem, our algorithm performs gradient descent using the gradient obtained from the approximate linear model until backtracking line search fails, upon which the model-based gradient is compared with the exact gradient obtained from a model-free algorithm. The difference between the model gradient and the exact gradient is then used for compensating future gradient-based updates. Our algorithm is shown to decrease the number of function evaluations compared with traditional model-free methods both in theory and in practice.
BibTeX
@misc{li2024accelerating,
  author = {Yansong Li and Shuo Han},
  title = {{Accelerating Model-Free Policy Optimization Using Model-Based Gradient: A Composite Optimization Perspective}},
  year = {2024},
  url = {https://openreview.net/forum?id=XcK4yCnLyJ},
  howpublished = {OpenReview},
}
Figure