Guide to Control: Offline Hierarchical Reinforcement Learning Using Subgoal Generation for Long-Horizon and Sparse-Reward Tasks

Wonchul Shin, Yusung KimIJCAI 2023 · DOI

TL;DR

Guider is a new offline hierarchical reinforcement learning (HRL) framework designed to conquer long-horizon tasks with sparse rewards. By pre-training a latent variable model to understand which subgoals are "reachable" from the existing data, Guider allows a high-level policy to generate realistic stepping stones that guide a low-level policy to the final goal, significantly outperforming existing methods on complex benchmarks like Antmaze-large.

What Problem Does This Paper Address?

Reinforcement Learning (RL) usually thrives on "trial and error," but in the real world, "error" can be expensive or dangerous. Offline RL attempts to learn optimal behavior from fixed, pre-recorded datasets without any environment interaction.

However, offline RL faces two massive hurdles:

  1. Long Horizons & Sparse Rewards: If a robot only gets a reward at the very end of a 1,000-step task, it’s incredibly hard to figure out which early actions were actually helpful.
  2. Distributional Shift: If a policy tries to reach a state or perform an action not well-represented in the dataset, the "value" of that action is often wildly overestimated, leading the agent to fail.

While Hierarchical RL (breaking a big task into subgoals) is a known solution, applying it offline is tricky. How do you ensure the high-level policy doesn't command the low-level policy to reach a "hallucinated" subgoal that isn't actually reachable based on the data?

Methods: The Guider Framework

The authors introduce Guider (Guide to Control), which uses a multi-stage training process to ground hierarchical decisions in the available data.

The Guider architecture: (1) Pre-training the latent subgoal model and (2) Training high-level and low-level policies with goal relabeling.
  1. Latent Subgoal Pre-training: Before training any policies, a latent variable model (encoder $q_\psi$, decoder $p_\theta$, and a prior $\rho_\omega$) is trained on the offline dataset. This model learns to map pairs of states to a latent space representing "reachability" within a short window (e.g., $c$ steps).
  2. High-Level Subgoal Generation: The high-level policy $\pi_h$ learns to pick a latent subgoal $z$ that maximizes the long-term reward. Crucially, this policy is KL-regularized against the pre-trained prior $\rho_\omega(z|s_t^G)$. This forces the high-level policy to only suggest subgoals that the dataset proves are physically reachable from the current state.
  3. Low-Level Subgoal Reaching: The low-level policy $\pi_l$ is trained to reach these subgoals using advantage-weighted supervised learning. It uses "hindsight goal relabeling"—looking at where it actually ended up and pretending that was the goal all along—to learn efficiently from sub-optimal data.

Key Insights and Ideas

  • The Power of the Prior: You don't need the environment to tell you if a subgoal is reachable if you have a well-trained latent prior. By constraining the high-level policy to the "latent distribution of reachability," Guider effectively prevents the agent from attempting impossible out-of-distribution maneuvers.
  • Stitching via Hierarchy: Guider excels at "stitching." In complex environments like Antmaze-large, the dataset might contain many random paths but few complete ones. Guider’s high-level policy learns to pick the "best" sequence of reachable points to create a new, successful path that never existed in the original data.
Visualization of the hierarchical process: the high-level policy generates a sequence of subgoals (stars) to guide the agent through the maze to the final flag.
  • Simplicity at the Bottom: Because the high-level policy breaks the task into easy, short-horizon chunks, the low-level policy doesn't need a complex RL algorithm. A simple weighted imitation of the data is enough to get the job done.

Key Results

Guider demonstrated state-of-the-art performance across standard D4RL benchmarks:

  • Antmaze-large: Guider achieved a 80.8% success rate, while the next best baseline (WGCSL) only managed 17.5%. Most standard offline RL methods (like CQL) fail completely here, scoring near 11%.
  • Kitchen-partial: It reached a 70.4% success rate, outperforming the skill-based hierarchical method OPAL (65.2%).
  • Ablation Studies: The authors proved that KL-regularization is the essential component. Without it, the success rate on Antmaze-large dropped from 80.8% to 57%, confirming that the latent prior is critical for keeping the high-level policy grounded in the data distribution.

Limitations and Future Work

  • Fixed Temporal Abstraction: Currently, the high-level policy generates a subgoal every $c$ steps (e.g., every 50 steps). An adaptive period—where the agent decides when it needs a new subgoal—could be more efficient.
  • State vs. Pixels: The current work focuses on low-dimensional state inputs. Extending Guider to high-dimensional visual inputs (RGB images) would require more complex encoders and decoders.
  • Subgoal Decoding: While the decoder maps latent subgoals back to physical space for the low-level policy, training the low-level policy directly in latent space is a promising alternative for future research.

Summary updated

BibTeX
@inproceedings{shin2023guide,
  author = {Wonchul Shin and Yusung Kim},
  title = {{Guide to Control: Offline Hierarchical Reinforcement Learning Using Subgoal Generation for Long-Horizon and Sparse-Reward Tasks}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/469},
}
Figure