Progressive Horizon Curricula for Long Horizon Agents

This direction is merged into another direction: curriculum learning. Its last version stays readable and citable here.

Version 1 of 115 papers24 conceptsfrozen 2026-10-04

Reinforcement learning (RL) agents often struggle with "long-horizon" tasks—problems where a success or failure only occurs after hundreds or thousands of individual actions. When an agent is dropped into a complex environment with a large interaction budget from the start, it typically falls into a trap of aimless exploration. Because the reward is so far away, the "credit" for a success is spread thin across too many actions, leading to noisy gradients and training instability.

To solve this, researchers have converged on a shared strategy: Progressive Horizon Curricula. Instead of training on the full task immediately, the agent begins with a truncated version of the problem and gradually expands its operational horizon as it improves.

The Credit Assignment Bottleneck

In a long-horizon task, like navigating a web browser to book a flight or solving a multi-step math proof, the number of possible action sequences grows exponentially with the number of steps. If an agent takes 100 steps and finally receives a reward, identifying which of those 100 steps were actually responsible for the success is a massive "credit assignment" problem.

Junhong Shen, Hao Bai, and their colleagues argue in Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction that training on long horizons from the start leads to "noisy credit assignment." The agent becomes lost in the noise of its own exploration. By restricting the agent to a short interaction budget initially, the model is forced to master direct, exploitative paths before it is allowed to attempt more complex, multi-step strategies.

Start Small to Learn Fast

The core mechanism of this research direction is the incremental expansion of the "episode length" or "interaction budget."

In the context of evolution strategies, Lior Fuks and Frank Hutter introduced Progressive Episode Lengths (PEL) in An Evolution Strategy with Progressive Episode Lengths for Playing Games. By starting with truncated versions of Atari games, their agents could rapidly explore the parameter space and learn fundamental behaviors—like moving toward a goal—before having to worry about long-term survival.

This same intuition applies to creative and physical tasks. In PaintBot: A Reinforcement Learning Approach for Natural Media Painting, Biao Jia and Chen Fang used a curriculum that increased the number of allowed brush strokes only after the agent met specific reward thresholds. This ensured the agent mastered basic stroke placement before attempting to compose a full, complex image.

Scaling Interaction for LLM Agents

The most recent application of this idea is in Large Language Model (LLM) agents. While much of the field focuses on "scaling thought" (giving models more tokens to reason), researchers are finding that "scaling interaction" (giving models more steps in an environment) is equally vital.

Zhiheng Xi, Jixuan Huang, and the team behind AgentGym-RL developed a strategy called ScalingInter-RL. They found that providing a large interaction budget at the start of training often caused model collapse. By starting with a limited budget and monotonically increasing the horizon, they enabled a 7B-parameter model to outperform much larger proprietary models like GPT-4o on multi-turn tasks.

This "staged" approach serves as a stabilizer. As Xiwen Liang and Bingqian Lin demonstrate in Structured Preference Optimization for Vision-Language Long-Horizon Task Planning, transitioning from short-horizon sequences to plans involving dozens of steps prevents the training instability that occurs when models are immediately exposed to high-complexity environments.

Mastery Across Domains

The success of progressive horizons across diverse fields suggests it is a fundamental principle for training autonomous systems:

By starting with a "shallow" version of the world, these agents build a foundation of basic skills. As the horizon recedes, they don't have to relearn how to act; they only have to learn how to chain their existing skills into longer, more sophisticated sequences.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_progressive_horizon_curricula_for_long_h_0a7dc72f,
  title = {Progressive Horizon Curricula for Long Horizon Agents},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/progressive-horizon-curricula-for-long-horizon-agents/rdr_0a7dc72f849c4ee889cf9250083d7468},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_progressive_horizon_curricula_for_long_h_0a7dc72f_v1,
  title = {Progressive Horizon Curricula for Long Horizon Agents},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/progressive-horizon-curricula-for-long-horizon-agents/rdr_0a7dc72f849c4ee889cf9250083d7468/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}