Horizon Contraction via Temporally Abstracted Actions

396 concepts263 papers5 proposals

When an autonomous agent operates in the real world, it faces a fundamental mismatch in timescales. A robot arm might require motor commands at 100 Hz, meaning a simple 10-second task like opening a jar requires 1,000 sequential decisions. If a planning algorithm attempts to search through this space step-by-step, it encounters an exponential explosion of possibilities. This is the granularity trap: reasoning at every millisecond is computationally prohibitive and makes long-term planning nearly impossible.

Furthermore, fine-grained control severely complicates credit assignment. If an agent receives a reward only after successfully opening the jar, propagating that reward signal back through thousands of individual motor adjustments is highly inefficient.

The core solution to this problem is temporal abstraction—grouping sequences of primitive actions into high-level, multi-step units (variously called macro-actions, options, behavior primitives, or latent plans). By changing the granularity of action, agents can contract the effective planning horizon, transforming a long-horizon problem into a short, manageable sequence of decisions.

Compressing Time through Horizon Contraction

The primary mechanism behind temporal abstraction is horizon contraction. Instead of forcing a policy or value function to model every transition, multiple primitive actions are compressed into a single logical step.

As Ahn et al. describe, this representational strategy simplifies value estimation by decreasing the number of temporal transitions the agent must account for to reach a goal. Instead of deciding what to do at millisecond 452, the agent commits to a high-level "behavior primitive" like grasping or pushing, as explored by Nasiriany et al..

Primitive Action Space (Long Horizon):
S0 -> a0 -> S1 -> a1 -> S2 -> a2 -> S3 -> a3 -> S4 -> a4 -> S5 (Reward!)

Temporally Abstracted Space (Contracted Horizon):
S0 ---------------------> Macro-Action 1 ---------------------> S5 (Reward!)

By evaluating rewards across larger temporal distances in fewer steps, value functions converge much faster. Theocharous et al. demonstrate that grouping primitive actions into temporally extended macro-actions transforms a standard decision process into a semi-Markov decision process (SMDP), accelerating reward propagation and simplifying planning.

Pruning the Search Tree

Why does horizon contraction make online planning computationally tractable? Consider a standard search tree with a branching factor $B$ (the number of available actions) and a depth $T$ (the planning horizon). The computational complexity of an exhaustive search scales exponentially as $O(B^T)$.

If an agent plans using primitive actions, $T$ is large, causing the search tree to explode. Temporal abstraction bypasses this combinatorial explosion by drastically reducing $T$. Lim et al. show that integrating macro-actions into Monte Carlo planning frameworks reduces the depth of the search tree, allowing solvers to handle significantly longer planning horizons in partially observable environments.

Similarly, Czechowski et al. demonstrate that planning over a graph where edges represent multi-step "milestones" rather than atomic actions allows agents to bypass intermediate details and focus entirely on high-level reasoning.

Mitigating Compounding Errors in Latent Spaces

Early formulations of temporal abstraction relied on pre-defined, hand-crafted macro-actions, which Mann et al. note can limit an agent's autonomy and flexibility. Modern deep reinforcement learning instead focuses on learning continuous or discrete latent skill spaces directly from data.

For instance, Jiang uses a state-conditional VQ-VAE to compress multi-step trajectories into single discrete latent codes. This mapping effectively shortens the planning horizon, enabling efficient long-term planning without the computational burden of high-frequency decision-making. Xie et al. achieve a similar effect by maintaining a selected latent skill for a fixed duration of multiple environment steps, preventing exponential search tree growth during online look-ahead searches.

Operating over these abstract latent plans is especially vital in offline reinforcement learning and world modeling. When an agent predicts the future step-by-step, small errors in its dynamics model accumulate rapidly over time, causing the predicted trajectory to drift away from reality. By operating over temporally extended latent plans instead of raw actions, Rosete-Beas et al. mitigate this compounding error problem, drastically decreasing the number of decision steps required to complete a task and stabilizing long-term predictions.

The Synergy of State and Temporal Abstraction

Temporal abstraction rarely acts in isolation; its benefits are amplified when paired with state abstraction. While temporal abstraction groups actions over time, state abstraction simplifies the representation of the environment itself, stripping away irrelevant features.

Konidaris et al. argue that combining both forms of abstraction provides a significant performance advantage over using temporal abstraction alone. While skills reduce the temporal horizon, state abstraction prevents the agent from becoming overwhelmed by high-dimensional sensory data during high-level planning.

Zhang et al. demonstrate this synergy by training agents to plan over discrete, human-interpretable "attributes" rather than raw sensory data. By focusing on transitions between discrete properties, the agent can compose learned skills at test time to achieve entirely novel, long-term goals.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_temporal_abstraction_0c93dcd2,
  title = {temporal abstraction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-abstraction/rdr_0c93dcd20c7a42b1b245ff0d37e4896d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_temporal_abstraction_0c93dcd2_v1,
  title = {temporal abstraction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-abstraction/rdr_0c93dcd20c7a42b1b245ff0d37e4896d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake