PlanDQ: Hierarchical Plan Orchestration via D-Conductor and Q-Performer
Lists
Only you see this
Abstract
Despite the recent advancements in offline RL, no unified algorithm could achieve superior performance across a broad range of tasks. Offline \textit{value function learning}, in particular, struggles with sparse-reward, long-horizon tasks due to the difficulty of solving credit assignment and extrapolation errors that accumulates as the horizon of the task grows.~On the other hand, models that can perform well in long-horizon tasks are designed specifically for goal-conditioned tasks, which commonly perform worse than value function learning methods on short-horizon, dense-reward scenarios. To bridge this gap, we propose a hierarchical planner designed for offline RL called PlanDQ. PlanDQ incorporates a diffusion-based planner at the high level, named D-Conductor, which guides the low-level policy through sub-goals. At the low level, we used a Q-learning based approach called the Q-Performer to accomplish these sub-goals. Our experimental results suggest that PlanDQ can achieve superior or competitive performance on D4RL continuous control benchmark tasks as well as AntMaze, Kitchen, and Calvin as long-horizon tasks.
BibTeX
@inproceedings{chen2024plandq,
author = {Chang Chen and Junyeob Baek and Fei Deng and Kenji Kawaguchi and Caglar Gulcehre and Sungjin Ahn},
title = {{PlanDQ: Hierarchical Plan Orchestration via D-Conductor and Q-Performer}},
booktitle = {International Conference on Machine Learning},
year = {2024},
url = {https://openreview.net/forum?id=17ZwoHl65h},
}| Simple Hierarchical Planning with Diffusion | ICLR · 2024 |
| Hierarchical Diffusion for Offline Decision Making | ICML · 2023 |
| IQL-TD-MPC: Implicit Q-Learning for Hierarchical Model Predictive Control | Frontiers4LCD @ ICML · 2023 |
| Diffused Task-Agnostic Milestone Planner | arXiv · 2023 |
| Efficient Planning with Latent Diffusion | arXiv · 2023 |
| Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning | arXiv · 2022 |