Overcoming Rigid Temporal Horizons in Hierarchical Reinforcement Learning

22 concepts21 papers1 proposal

To solve complex, long-horizon tasks, intelligent agents cannot simply plan at the level of individual motor commands. Instead, they must use temporal abstraction—grouping sequences of low-level actions into reusable, high-level behaviors often called "skills," "subroutines," or "primitives."

In traditional Hierarchical Reinforcement Learning (HRL), a high-level policy orchestrates behavior by selecting a skill or generating a subgoal, which a low-level policy then executes. To make this interaction mathematically tractable, early and foundational frameworks introduced a major simplifying assumption: fixed-duration temporal abstraction. In these systems, every chosen skill or subgoal is executed for a predetermined, uniform number of timesteps ($H$) before control returns to the high-level policy.

While computationally convenient, this lock-step assumption introduces a fundamental flaw. Real-world tasks do not unfold in uniform blocks of time. Forcing an agent to operate within rigid temporal horizons creates a severe mismatch between the artificial constraints of the architecture and the natural rhythms of the environment.

The Semantic Mismatch of Fixed Horizons

To understand why fixed temporal horizons fail, consider a robot tasked with navigating a house to fetch a cup. This task naturally breaks down into sub-tasks of wildly different durations:

  1. Locate and walk down a long corridor (hundreds of timesteps).
  2. Turn a door handle (a few timesteps).
  3. Grasp a cup (a single, precise moment).

If an HRL framework enforces a rigid horizon of $H = 20$ timesteps across all skills, the agent faces a destructive dilemma. For brief actions like turning a handle or grasping an object, the skill finishes its semantic goal long before the 20-step window expires. The low-level policy is forced to either pad the remaining time with idle behavior or drift into unintended actions. Conversely, for extended actions like navigating a corridor, 20 steps are vastly insufficient. The high-level policy is forced to repeatedly re-select the same navigation skill over and over again, defeating the very purpose of temporal abstraction, which is to shorten the effective planning horizon for the high-level controller.

This structural limitation was recognized early on in hierarchical goal recognition. Nate Blaylock et al. in Fast Hierarchical Goal Schema Recognition noted that enforcing simultaneous transitions across hierarchical levels fails to capture the varying temporal durations of different subgoals, relying instead on a rigid Markovian approximation at every level.

When agents are forced to learn skills under these static constraints, the learned behaviors lose their semantic clarity. As Kevin Frans et al. demonstrated in Meta Learning Shared Hierarchies (MLSH), assuming a fixed timescale for sub-policy execution assumes all motor primitives require the same amount of time to complete. This rigidity fails in complex environments where different skills—such as jumping versus walking—naturally occur over varying durations. Similarly, Sainbayar Sukhbaatar et al. found in Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning that static temporal constraints restrict the flexibility of learned hierarchies, preventing agents from dynamically adjusting sub-task durations based on inherent task complexity.

The Cost of Rigidity in Representation and Planning

The consequences of fixed temporal horizons extend beyond execution inefficiencies; they fundamentally cripple an agent's ability to learn clean representations from data.

When extracting skills from unstructured offline datasets or human demonstrations, behavior must be segmented. If a framework uses fixed-length temporal windows to slice this data, it cuts across natural behavioral boundaries. Ashish Kumar et al. showed in Learning Navigation Subroutines from Egocentric Videos that fixed-length segmentation fails to capture the natural duration of complex tasks, arguing that effective control requires variable-length subroutines that adapt to specific action requirements.

Furthermore, pre-determining the temporal horizon limits the expressiveness of skill priors. Karl Pertsch et al. argued in Accelerating Reinforcement Learning with Learned Skill Priors that fixed time horizons prevent models from learning flexible, semantic skills that vary in length based on environmental demands. This sentiment is echoed across offline reinforcement learning paradigms:

Real-World Failure Modes: Driving and Robotics

The limitations of fixed-duration abstractions become acute in real-world applications like autonomous driving and language-conditioned robotics, where temporal variability is a defining characteristic of the domain.

In autonomous driving, maneuvers like changing lanes, executing a turn, or coming to an emergency stop occur over highly disparate timescales. Hang Zhao 0021 et al. demonstrated in Boosting Offline Reinforcement Learning for Autonomous Driving with Hierarchical Latent Skills that fixed-duration abstractions fail to represent the temporal complexity of real-world driving, necessitating variable-length skill discovery to handle diverse maneuvers safely and smoothly.

Similarly, in language-conditioned robotics, human instructions imply multi-step plans with highly variable durations (e.g., "open the drawer, take out the knife, and slice the tomato"). Haobin Jiang et al. observed in Discrete Latent Plans via Semantic Skill Abstractions that relying on a fixed temporal horizon for every discrete skill prevents models from efficiently handling the natural variability between brief actions like grasping and extended actions like opening a door.

The Shift Toward Variable-Duration Hierarchies

To overcome these limitations, recent research has shifted away from rigid, human-defined temporal windows toward frameworks that dynamically determine skill boundaries and integrate multi-scale temporal rhythms.

One approach focuses on learning world models that explicitly decouple planning from fixed time steps. Benjamin Freed et al. showed in Learning Temporally AbstractWorld Models without Online Experimentation that frameworks must dynamically determine the start and end points of learned skills to remain robust when environmental or physical properties change. In offline planning, Wonchul Shin et al. demonstrated in Guide to Control: Offline Hierarchical Reinforcement Learning Using Subgoal Generation for Long-Horizon and Sparse-Reward Tasks that replacing static intervals with an adaptive mechanism for issuing subgoals significantly improves efficiency in dynamic environments.

Another powerful trend leverages generative diffusion models to plan across fluid timescales. While early hierarchical diffusion models still suffered from rigid horizons, the authors of SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Execution identified the implementation of variable-length horizons as a critical future direction to allow for flexible temporal scaling across diverse robotic actions.

Breaking this bottleneck completely, Ce Hao introduced CHD: Coupled Hierarchical Diffusion for Long-Horizon Tasks, a framework that treats subgoal selection and trajectory generation as a single, unified diffusion process. By allowing low-level feedback to refine high-level plans in real-time, the system accommodates variable-length segments, seamlessly transitioning between rapid manipulations and extended navigation phases.

Fixed-Duration Hierarchy (Rigid & Inefficient):
High-Level:  [ Skill A (20 steps) ] -> [ Skill B (20 steps) ] -> [ Skill C (20 steps) ]
Low-Level:   [--Actual Task A--][Pad]   [--Cut Off Task B early--] [---Task C Continues---]

Variable-Duration Hierarchy (Adaptive & Semantic):
High-Level:  [ Skill A (12 steps) ] -> [ Skill B (28 steps) ] -> [ Skill C (20 steps) ]
Low-Level:   [--Actual Task A--]       [------Actual Task B------] [----Actual Task C----]

Finally, moving entirely away from human-defined temporal structures, Xianghua Zeng et al. developed Structural Information-based Hierarchical Diffusion for Offline Reinforcement Learning. Instead of imposing static temporal windows, their approach uses graph-based structural information to adaptively cluster states. This allows the hierarchy to naturally capture the multi-scale complexity and varying temporal rhythms inherent in real-world tasks.

By replacing rigid, clock-driven intervals with event-driven, variable-duration abstractions, modern hierarchical systems allow agents to plan over extended horizons without sacrificing the precision needed for immediate, fine-grained control.

Key Foundational Work

META LEARNING SHARED HIERARCHIES (2018) — contributed 1 concept elements to this direction.
Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning (2018) — contributed 1 concept elements to this direction.
Fast Hierarchical Goal Schema Recognition (2006) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_fixed_duration_temporal_abstraction_deb0480d,
  title = {fixed-duration temporal abstraction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fixed-duration-temporal-abstraction/rdr_deb0480de3974fedab118e19f61e8d42},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_fixed_duration_temporal_abstraction_deb0480d_v1,
  title = {fixed-duration temporal abstraction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fixed-duration-temporal-abstraction/rdr_deb0480de3974fedab118e19f61e8d42/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake