Temporal Action Abstraction for Long Horizon Tasks

This direction is merged into another direction: temporal abstraction. Its last version stays readable and citable here.

Version 1 of 1277 papers337 conceptsfrozen 2026-10-04

Reinforcement learning (RL) often fails when the gap between an action and its reward is too large. In a "flat" RL setup, an agent must learn to associate a reward received at step 1,000 with a specific muscle twitch at step 1. This is the credit assignment problem, and it becomes mathematically intractable as the "horizon"—the number of steps required to complete a task—grows.

Temporal action abstraction solves this by shrinking the effective horizon. Instead of deciding what to do at every millisecond, the agent learns to think in "chunks" of time. By grouping sequences of primitive actions into high-level skills or macro-actions, the agent reduces a thousand-step problem into a ten-step problem.

The Hierarchical Split: Managers and Workers

The most common way to implement temporal abstraction is through a two-tier hierarchy. This architecture, popularized by Tejas D. Kulkarni and colleagues in Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation, separates the "what" from the "how."

  1. The Meta-Controller (Manager): Operates at a coarse temporal scale. It looks at the state and selects a high-level subgoal or skill (e.g., "go to the door").
  2. The Controller (Worker): Operates at a fine temporal scale. It receives the subgoal from the manager and executes the sequence of primitive actions (e.g., specific motor torques) required to reach it.

This separation allows the manager to ignore the messy details of motor control and focus on long-term strategy. As shown in A Deep Hierarchical Approach to Lifelong Learning in Minecraft by Chen Tessler and others, this hierarchy is essential for environments like Minecraft, where a single high-level goal (like "find a diamond") requires thousands of low-level interactions.

Action Chunking and Tokenization

While subgoals define where to go, macro-actions define how to move in blocks. Recent work has moved toward treating robot actions like words in a sentence.

In PRISE: LLM-Style Sequence Compression for Learning Temporal Action Abstractions in Control, Ruijie Zheng and Ching-An Cheng apply Byte Pair Encoding (BPE)—the same algorithm used to tokenize text for Large Language Models—to continuous robot actions. By identifying frequently occurring sequences of actions and "tokenizing" them into single macro-actions, the agent can reason about complex manipulation tasks as short sequences of tokens rather than long streams of numbers.

Similarly, the DEAS framework by Changyeon Kim and colleagues treats fixed-length action sequences as single temporal units. This "chunking" reduces the search depth for planning algorithms, allowing models to scale to the complexity of Vision-Language-Action (VLA) robotics.

Planning in the Latent Skill Space

If we can compress actions into skills, we can also plan in that compressed space. Standard planning algorithms like Monte Carlo Tree Search (MCTS) struggle in high-dimensional environments because the branching factor is too high.

Baiting Luo and others addressed this in Scalable Decision-Making in Stochastic Environments through Learned Temporal Abstraction by learning discrete macro-actions via a VQ-VAE. By searching through a tree of high-level macro-actions rather than low-level steps, their L-MAP planner achieves significantly lower latency.

This concept of "latent planning" is pushed further in Latent Skill Planning for Exploration and Transfer by Kevin Xie and Danijar Hafner. By mapping high-level decisions to sequences of environment steps, the agent can "look ahead" much further into the future with the same amount of computation.

Horizon Contraction: Why the Math Works

The fundamental intuition behind all these methods is horizon contraction. When you group $N$ steps into one abstract action, you effectively divide the task's difficulty by $N$.

In Learning to Reason as Action Abstractions with Scalable Mid-Training RL, Shenao Zhang and colleagues demonstrate that this applies even to LLM reasoning. By grouping individual tokens into "reasoning blocks," the model experiences a shorter planning horizon, which leads to faster convergence during reinforcement learning.

This contraction also simplifies safety. As noted in Skill-based Safe Reinforcement Learning with Risk Planning, planning over extended skills allows an agent to "see" risks that only manifest after several steps—such as a robot losing its balance—which would be invisible to a policy looking only one step ahead.

Summary of Benefits

Temporal action abstraction transforms the reinforcement learning landscape by:

  • Reducing Search Depth: Planning algorithms can reach distant goals with fewer "jumps" (L-MAP).
  • Improving Credit Assignment: Rewards are linked to high-level choices, not just low-level noise (Option-aware Temporally Abstracted Value).
  • Enabling Reusability: Once a "skill" (like opening a door) is learned, it can be reused across different tasks (BUDS).

By moving from "muscle twitches" to "meaningful movements," agents can finally tackle the long-horizon complexity of the real world.

Papers in this version

Showing 60 of 277 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_temporal_action_abstraction_for_long_hor_3e95f96e,
  title = {Temporal Action Abstraction for Long Horizon Tasks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-action-abstraction-for-long-horizon-tasks/rdr_3e95f96e4e0545868f3637542fe0b3dd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_temporal_action_abstraction_for_long_hor_3e95f96e_v1,
  title = {Temporal Action Abstraction for Long Horizon Tasks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-action-abstraction-for-long-horizon-tasks/rdr_3e95f96e4e0545868f3637542fe0b3dd/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}