Making Better Decision by Directly Planning in Continuous Control

Jinhua Zhu, Yue Wang, Lijun Wu, Tao Qin, Wengang Zhou, Tie-Yan Liu, Houqiang LiICLR 2023 · OpenReview

TL;DR

The authors introduce POMP (Policy Optimization with Model Planning), a model-based reinforcement learning algorithm that uses a novel gradient-based planner called D3P to optimize action sequences. By leveraging first-order Taylor expansions of the Bellman equation, POMP achieves state-of-the-art sample efficiency and performance on complex continuous control tasks like MuJoCo's Humanoid.

What Problem Does This Paper Address?

In reinforcement learning (RL), model-based methods (MBRL) are prized for their sample efficiency—they learn a "world model" to simulate the environment. However, most MBRL methods simply use this model to generate synthetic data for training a policy network.

The success of systems like AlphaZero and MuZero suggests that directly planning with the model at decision time is much more effective. But there’s a catch: planning in continuous action spaces is incredibly difficult. Unlike discrete games (like Chess or Go) where you can search a finite tree of moves, continuous spaces have an infinite number of candidate actions. Furthermore, actions are temporally dependent—an update to an action at step one changes the state at step two, making the entire trajectory a moving target for optimization.

Methods

The paper proposes two interconnected components:

  1. D3P (Deep Differential Dynamic Programming): This is the core planner. Inspired by classical optimal control, it treats planning as an optimization problem.
  2. Local Quadratic Programming: To avoid the "infinite action" problem, D3P uses a first-order Taylor expansion of the Bellman equation to construct a local quadratic objective. This allows the agent to find the optimal local action update analytically, without needing expensive second-order derivatives (Hessians).
  3. Feedback Term: To handle temporal dependencies, D3P incorporates a feedback term. When an action at a previous timestep is updated, the planner adjusts subsequent actions based on the predicted change in state, rather than trying to update all actions simultaneously in a vacuum.
  4. POMP (Policy Optimization with Model Planning): This is the overall RL framework. It integrates D3P into existing MBRL architectures (like MAAC or MBPO).
  5. Smart Initialization: Instead of starting the planner from scratch, it uses the current policy network to provide a "warm start" action sequence.
  6. Conservative Planning: To prevent the planner from exploiting errors in the learned model (a common failure mode in MBRL), the authors add a conservative term (log-likelihood of the action under the policy) to keep the plan near the training data distribution.

Datasets

The researchers evaluated their method on the MuJoCo benchmark suite, a standard for continuous control. The tasks included:

  • Low-dimensional: InvertedPendulum, Hopper, Walker2d.
  • High-dimensional/Complex: HalfCheetah, Ant, and Humanoid.

Key Insights and Ideas

  • Planning > More Policy Updates: The authors found that simply increasing the number of times you update a policy network (gradient steps) doesn't necessarily lead to better performance. However, adding a planning step consistently improves the decision-making quality.
  • The "Warm Start" is Essential: Initializing the D3P planner with a random action sequence fails completely. The policy network provides the necessary "common sense" starting point for the planner to refine.
  • Efficiency of First-Order Methods: D3P significantly outperforms zero-order planning methods like the Cross-Entropy Method (CEM) or random shooting. By using the gradient information from the model, it finds better trajectories with fewer iterations.
Comparison of different planners
  • Model Quality vs. Planning: Interestingly, the D3P planner can improve performance even with "early-stage" models that haven't been trained on much data, suggesting that the planning signal is robust.

Key Results

  • State-of-the-Art Performance: POMP consistently outperformed strong baselines including SAC (model-free), MBPO, and MAGE (model-based) across all MuJoCo tasks.
Learning curves on MuJoCo tasks
  • High-Dimensional Success: The gains were most pronounced in the most difficult tasks. In Humanoid (the most complex MuJoCo environment), POMP achieved a significantly higher final return than all other methods.
  • Convergence: The authors provided a theoretical proof for the convergence rate of the D3P planner, showing it achieves at least linear convergence under mild assumptions.

Limitations and Future Work

While POMP is highly effective, it still relies on the accuracy of the learned world model. If the model has significant bias or "blind spots," the planner might still be misled, despite the conservative term.

The authors suggest that future work should focus on model uncertainty. By explicitly estimating how "sure" the model is about a certain state-action pair, the planner could more effectively trade off between exploring new areas and exploiting known high-reward trajectories. Additionally, combining D3P with principled discrete search methods (like UCT) for hybrid domains remains an open research direction.

Summary updated

Abstract

By properly utilizing the learned environment model, model-based reinforcement learning methods can improve the sample efficiency for decision-making problems. Beyond using the learned environment model to train a policy, the success of MCTS-based methods shows that directly incorporating the learned environment model as a planner to make decisions might be more effective. However, when action space is of high dimension and continuous, directly planning according to the learned model is costly and non-trivial. Because of two challenges: (1) the infinite number of candidate actions and (2) the temporal dependency between actions in different timesteps. To address these challenges, inspired by Differential Dynamic Programming (DDP) in optimal control theory, we design a novel Policy Optimization with Model Planning (POMP) algorithm, which incorporates a carefully designed Deep Differential Dynamic Programming (D3P) planner into the model-based RL framework. In D3P planner, (1) to effectively plan in the continuous action space, we construct a locally quadratic programming problem that uses a gradient-based optimization process to replace search. (2) To take the temporal dependency of actions at different timesteps into account, we leverage the updated and latest actions of previous timesteps (i.e., step $1, \cdots, h-1$) to update the action of the current step (i.e., step $h$), instead of updating all actions simultaneously. We theoretically prove the convergence rate for our D3P planner and analyze the effect of the feedback term. In practice, to effectively apply the neural network based D3P planner in reinforcement learning, we leverage the policy network to initialize the action sequence and keep the action update conservative in the planning process. Experiments demonstrate that POMP consistently improves sample efficiency on widely used continuous control tasks. Our code is released at https://github.com/POMP-D3P/POMP-D3P.
BibTeX
@inproceedings{zhu2023making,
  author = {Jinhua Zhu and Yue Wang and Lijun Wu and Tao Qin and Wengang Zhou and Tie-Yan Liu and Houqiang Li},
  title = {{Making Better Decision by Directly Planning in Continuous Control}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=r8Mu7idxyF},
}
Figure