In reinforcement learning and robotics, the "curse of horizon" is a constant challenge. To solve complex tasks—like navigating a building or tidying a kitchen—an agent must plan across thousands of individual timesteps. Standard approaches handle this by "chunking" time, using fixed-length action sequences or subgoals to simplify the search space.
However, recent research suggests that these rigid temporal windows are a fundamental bottleneck. Whether it is a robot making a split-second adjustment to a grasp or a navigator following a long, straight hallway, the "natural" duration of an action is rarely constant. The field is moving toward adaptive temporal abstraction: systems that dynamically determine how far to look ahead and how long to commit to a behavior based on the immediate complexity of the task.
The Rigidity of Fixed Horizons
Most hierarchical models rely on a fixed hyperparameter for their planning horizon. For example, OptionZero extends the MuZero algorithm by planning with "options" (sequences of actions), but Po-Wei Huang and colleagues note that a static maximum length for these options limits an agent's ability to adapt to varying scales of abstraction.
This rigidity creates a "Goldilocks" problem. If the horizon is too short, the agent lacks the foresight to coordinate long-term behavior, as seen in multi-agent systems where short horizons lead to unachievable goals Liangzhou Wang et al.. If the horizon is too long, the agent becomes unresponsive, unable to react to sudden environmental changes because it is "locked in" to a pre-planned sequence Improving planning and MBRL with temporally-extended actions.
Learning Variable-Length Primitives
To move beyond fixed windows, researchers are developing methods to discover "primitives" or "subroutines" that have natural, variable durations. Instead of segmenting a demonstration into equal 10-step chunks, models like HIPS Kalle Kujanpää and Joni Pajarinen decompose trajectories into variable-length segments based on task complexity. This allows a model to allocate more planning resources to a difficult maneuver (like threading a needle) while skipping over simple, predictable sequences (like moving an arm through empty space).
Similarly, the OPAL framework OPAL and DEAS Changyeon Kim et al. highlight that static sequence lengths fail to capture the difference between long-term locomotion and short-term object interaction. By treating action sequences as "chunks" that can be detached from the value learning process, these models can scale to much longer horizons without losing the granularity needed for precise control.
Dynamic Planning and Mixture of Horizons
If the environment's demands change, the agent's internal "clock" should change with them. A recent breakthrough in Vision-Language-Action (VLA) models is the Mixture of Horizons (MoH) Mixture of Horizons in Action Chunking. Rather than picking one horizon, MoH predicts multiple horizons simultaneously and uses a learned gating mechanism to fuse them. This allows the robot to maintain a broad strategic context while simultaneously attending to the fine-grained details required for delicate manipulation.
This concept of "temporal attention" is also explored in GPT2-DTMA Linjie Xu et al.. By identifying "Markov heads" in pretrained transformers—heads that focus only on the most recent token—the researchers developed a mechanism to adaptively weight short-term reactive control against long-term planning. This allows a single architecture to excel at both "twitch" tasks like walking and "strategic" tasks like navigation.
State-Dependent Termination
The ultimate goal of adaptive abstraction is to move from "clock-based" control to "event-based" control. In traditional Hierarchical Reinforcement Learning (HRL), a high-level policy might pick a skill and run it for exactly $N$ steps. Carlos Florensa, Yan Duan, and Pieter Abbeel argued early on that this lacks flexibility; instead, skills should conclude based on environmental state—a "learned termination signal."
We see this intuition realized in frameworks like PEAR Utsav Singh and Vinay P. Namboodiri, which adaptively relabels subgoals based on the agent's actual progress rather than a fixed timer. In industrial planning, Giuseppe Della Penna et al. use "duration inequalities" to let a planner determine the optimal termination point for a continuous process.
By allowing the state of the world to dictate the duration of an action, these systems achieve a level of reactivity that fixed-horizon models cannot match. As Kuo-Hao Ho demonstrates with Macro Action Quantization, constraining agents to human-like, variable-duration "macro actions" not only improves performance but also results in smoother, more predictable behavior that mirrors how humans naturally segment their own actions.