Grounding Subgoal Reachability in Hierarchical Reinforcement Learning

51 concepts35 papers

Hierarchical Reinforcement Learning (HRL) solves complex, long-horizon tasks by dividing labor between two tiers: a high-level manager (or planner) that breaks a distant goal into intermediate subgoals, and a low-level worker (or controller) that executes primitive actions to reach those subgoals.

In theory, this division of labor makes massive search spaces manageable. In practice, HRL frequently suffers from a critical structural bottleneck: subgoal infeasibility. Because the manager operates at a higher level of temporal or symbolic abstraction, it easily loses touch with physical reality. It frequently "hallucinates" or proposes subgoals that are physically, temporally, or logically impossible for the worker to achieve.

When a manager assigns an unreachable subgoal, the entire training loop collapses. The worker fails its task and receives no reward, while the manager receives a noisy, uninformative feedback signal because it cannot distinguish between a poor high-level strategy and a low-level execution failure Guo et al., Chakraborty et al.. To make HRL work, the manager's imagination must be strictly grounded in the worker's actual capabilities.


Why Subgoals Become Infeasible

Subgoal infeasibility is driven by three primary friction points between the planning and execution layers:

  1. The Moving Target Problem (Non-Stationarity): HRL requires both layers to learn simultaneously. As the manager learns where it wants to go, the worker is still figuring out how to move. If the manager proposes a subgoal that requires a skill the worker hasn't mastered yet, the worker fails, stalling exploration for both levels Singh et al..
  2. Topological Blindness: High-level policies often evaluate distance using simple geometric metrics (like Euclidean distance) rather than the true layout of the environment. A subgoal might be physically close in coordinate space but separated by an impassable wall, making it functionally unreachable Precup et al..
  3. Value Function Hallucination: In offline HRL, where agents learn from static datasets, parametric planners struggle to estimate values accurately for out-of-distribution states. This leads them to propose useless or non-existent waypoints that are completely absent from the training data Ahn et al., Baek et al..

This constraint applies even to neuro-symbolic architectures; Mitchener notes that the success of high-level symbolic plans is fundamentally bounded by whether the underlying reactive controllers can reliably execute their assigned subgoals.


Strategy 1: Adjacency and Topological Constraints

The most direct way to prevent a manager from picking impossible targets is to mathematically restrict its search space to a region the worker can realistically navigate.

Guo et al. and Guo et al. introduced the Hierarchical Reinforcement learning with Adjacency Constraint (HRAC) framework. HRAC restricts the manager to selecting subgoals within a defined $k$-step reachable neighborhood of the current state. By shrinking the target space, the worker is only tasked with achieving physically attainable objectives. To stabilize this interaction dynamically, frameworks like MENTOR introduce dynamic distance constraints that adjust as the agent learns.

Euclidean Planning (Blind)           Topological Planning (Grounded)
      [Manager Subgoal]                         [Subgoal]
            x                                       |
   =================== (Wall)              =================== (Wall)
            |                                       |
     [Worker State]                         [Worker]--->[Waypoint]

However, geometric proximity does not guarantee reachability. To handle complex obstacles, Precup et al. incorporate spatial information via graph representations, aligning subgoal generation with the environment's true topological constraints. By forcing the manager to respect the actual "lay of the land," the system avoids generating subgoals trapped behind walls or furniture.


Strategy 2: Grounding via Worker Value Functions

Instead of imposing rigid geometric boundaries, another class of solutions forces the manager to read the worker's "resume"—making the planner explicitly aware of the controller's current capabilities.

Levy et al. pioneered this with the Hierarchical Actor-Critic (HAC) framework. HAC periodically runs test transitions where the worker attempts to reach a subgoal using its current policy without any "hindsight" modifications. If the worker fails, the manager receives a direct penalty. Similarly, Gao et al. penalize the high-level policy when its chosen subgoals result in low-level collisions or timeouts, forcing the planner to respect the physical constraints of the agent.

Rather than penalizing failures after the fact, modern approaches use the worker's value function as a regularizer during the manager's decision-making process. Chakraborty et al. and Singh et al. developed the DIPPER framework, which integrates the lower-level agent's value function directly into the higher-level policy's objective function. Because a value function represents the expected success from a given state, this regularization ensures the manager only selects subgoals where the worker already has a high probability of execution.


Strategy 3: Generative Models and Graph Translators

When planning over massive distances, forcing a strict step-by-step constraint can limit exploration. To decouple long-term planning from short-term execution limits, researchers use generative models and graphs to "translate" distant goals into feasible steps.

Lee et al. proposed DHRL, which uses a graph as a translator to decompose distant, long-horizon subgoals into a sequence of short-horizon, reachable waypoints. In offline settings where data is sparse and unconstrained target spaces lead to noisy selections Shan et al., Shin et al. pre-train a latent variable model to capture the distribution of reachable states, ensuring the manager cannot propose subgoals absent from the dataset.

[Distant Goal] ---> (Graph Translator / Diffusion Model) ---> [Reachable Waypoint 1] ---> [Reachable Waypoint 2] ---> [Worker]

To dynamically match the worker's evolving skills, Pajarinen et al. introduced an adversarial framework (SAGA) where a GAN-like discriminator distinguishes between proposed subgoals and states actually achieved by the worker, forcing the manager to stay within the worker's current operational envelope.

Taking this further, Wang et al. utilize a conditional diffusion model regularized by a Gaussian Process prior, giving the manager a principled way to quantify uncertainty and avoid hallucinating targets in unexplored regions. Similarly, Haramati et al. leverage factored subgoal diffusion to generate intermediate steps that only change one or two object properties at a time, simplifying the reachability landscape for the worker.


Strategy 4: Inter-Skill Compatibility and Strict Execution

Even if individual subgoals are reachable, a chain of subgoals can still fail if the hand-off between consecutive skills is poorly coordinated.

Klimek et al. identified this as a state-distribution mismatch bottleneck: if a preceding skill terminates in a state outside the training distribution of the subsequent skill, execution fails. Szot et al. demonstrated this vividly in mobile manipulation tasks: an agent might successfully navigate to a cabinet (Sub-task 1), but position its physical body in a way that blocks the cabinet door from opening (Sub-task 2). The terminal state of the first skill is incompatible with the starting requirements of the next.

Incompatible Hand-off:
[Skill 1: Navigate to Door] ---> Terminal State: Robot blocks the door ---> [Skill 2: Open Door] (FAILED)

Compatible Hand-off:
[Skill 1: Navigate to Side] ---> Terminal State: Clear path to handle  ---> [Skill 2: Open Door] (SUCCESS)

To enforce rigorous coordination, Gupta et al. introduced a relay data-relabeling strategy to ensure smooth action hand-offs across sequences. To address this from an optimization perspective, Sun et al. propose a bidirectional reachability metric that incentivizes both levels to be mutually responsive.

Finally, Han et al. proposed Strict Subgoal Execution (SSE), which terminates training episodes immediately if the low-level worker fails to reach a designated subgoal. This "reach-it-or-fail" policy, paired with safety boundaries like those in the HiSaRL framework, provides an unambiguous negative signal that forces the high-level planner to prioritize reliable, compatible transitions over overly optimistic but unachievable paths.

Key Foundational Work

Learning Multi-Level Hierarchies with Hindsight (2019) — contributed 3 concept elements to this direction.
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (2019) — contributed 1 concept elements to this direction.
Hierarchical Reinforcement Learning with Parameters (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_subgoal_infeasibility_febac8d6,
  title = {subgoal infeasibility},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/subgoal-infeasibility/rdr_febac8d639bc4fd1b2030402e23290ed},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_subgoal_infeasibility_febac8d6_v1,
  title = {subgoal infeasibility},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/subgoal-infeasibility/rdr_febac8d639bc4fd1b2030402e23290ed/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake