Grounding Hierarchical Planning in Low-Level Reachability

This direction is split into other directions: subgoal infeasibility. Its last version stays readable and citable here.

Version 1 of 183 papers115 conceptsfrozen 2026-10-04

Hierarchical Reinforcement Learning (HRL) promises to solve long-horizon tasks by decomposing them into a "Manager" that sets abstract subgoals and a "Worker" that executes them. In theory, this should allow agents to plan across vast distances and complex sequences. In practice, these hierarchies often collapse.

The fundamental failure mode is a grounding gap: the Manager proposes subgoals that are theoretically sound but physically or computationally unreachable for the Worker. This creates a "moving target" problem where neither level can learn effectively because their interface—the subgoal—is disconnected from reality.

The Moving Target Problem

In a standard hierarchy, the Manager and Worker learn simultaneously. This creates a vicious cycle of non-stationarity. As the Worker improves its motor skills, the set of states it can reach changes. If the Manager learns a value function based on an old, incompetent version of the Worker, its plans become obsolete.

Joni Pajarinen and Vivienne Huiling Wang describe this as a lack of "principled uncertainty." Without knowing the Worker's current boundaries, the Manager "hallucinates" targets. To solve this, their SAGA framework (State-Conditioned Adversarial Subgoal Generation) introduces an adversarial discriminator. The discriminator's job is to distinguish between subgoals proposed by the Manager and states actually achieved by the Worker. This forces the Manager to only propose subgoals that "look like" something the Worker has successfully done, effectively tethering the high-level plan to the low-level's current skill set.

The Pitfalls of Lazy Planning

A common technique to speed up HRL is "hindsight relabeling," where a failed attempt to reach Subgoal A is relabeled as a successful attempt to reach Subgoal B (the state the agent actually ended up in). While this provides a dense training signal for the Worker, Ji-Hoon Hwang and colleagues argue in Strict Subgoal Execution that this creates "lazy planning."

When the Manager is never penalized for the Worker's failure, it learns that it can receive credit regardless of whether its specific intermediate targets are met. This leads to unreliable plans that fall apart during real-world execution. The solution proposed in SSE is a "reach-it-or-fail" policy: if the Worker fails to reach the exact subgoal, the episode terminates immediately. This provides a harsh but clear negative signal, forcing the Manager to prioritize reliability over optimism.

Grounding in Geometry and Topology

Sometimes the disconnect isn't about skill, but about the representation of the world. Karmesh Yadav and the creators of the FindingDory benchmark (FindingDory: A Benchmark to Evaluate Memory in Embodied Agents) found that high-level planners often select "goal frames" that are visually informative but lack the geometric features a low-level navigator needs.

To bridge this, researchers like Shuyuan Zhang and Doina Precup suggest incorporating spatial and topological constraints directly into the hierarchy (Incorporating Spatial Information into Goal-Conditioned Hierarchical Reinforcement Learning via Graph Representations). By using a graph to represent the environment, the Manager is physically restricted from proposing subgoals that are "near" in Euclidean space (like a point on the other side of a wall) but "far" in functional reachability.

Bi-Level Alignment and Cooperation

The most recent trend moves away from treating the Manager as a superior and the Worker as a subordinate, instead viewing them as a cooperative multi-agent system.

Why It Works: The Intuition of Feasibility

The core intuition across these 75 papers is that abstraction is only useful if it is grounded. A plan to "go to the kitchen" is useless to a robot that doesn't know how to turn its wheels. By implementing reachability constraints—whether through adversarial discriminators (SAGA), strict execution penalties (SSE), or graph-based bottlenecks (HRAC) — we ensure that the Manager's search space is a subset of the Worker's capability space. When the high-level "why" is constrained by the low-level "how," the hierarchy becomes a stable, unified agent capable of solving tasks that are otherwise too long or sparse to learn.

Papers in this version

Showing 60 of 83 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_grounding_hierarchical_planning_in_low_l_e9f4cd19,
  title = {Grounding Hierarchical Planning in Low-Level Reachability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/grounding-hierarchical-planning-in-low-level-reachability/rdr_e9f4cd1921b44ccb80618481dcda7799},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_grounding_hierarchical_planning_in_low_l_e9f4cd19_v1,
  title = {Grounding Hierarchical Planning in Low-Level Reachability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/grounding-hierarchical-planning-in-low-level-reachability/rdr_e9f4cd1921b44ccb80618481dcda7799/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}