Automated Curriculum Generation for Sparse Reward Environments

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2151 papers160 conceptsfrozen 2026-10-04

Reinforcement learning (RL) has a well-known "kryptonite": sparse rewards. If a robot only receives a signal when it perfectly completes a complex task—like stacking five blocks or navigating a high-traffic intersection—it will likely spend an eternity flailing randomly before ever seeing a +1.

Traditionally, researchers solved this by manually designing a "curriculum"—a sequence of increasingly difficult tasks that guide the agent from simple behaviors to complex mastery. However, as Vaibhav Bajaj and Peter Stone note in Task Phasing, hand-crafting these sequences is labor-intensive, introduces human bias, and often fails to adapt to the agent's actual learning bottlenecks. The research direction of Automated Curriculum Learning (ACL) seeks to remove the human from the loop, allowing agents to autonomously discover the "stepping stones" required to solve the target task.

The Goldilocks Zone of Difficulty

The core intuition behind ACL is finding the "Goldilocks zone": tasks that are neither too easy (where the agent learns nothing new) nor too hard (where the agent never succeeds).

David Held and Carlos Florensa formalized this in Goal GAN by using a Generative Adversarial Network to propose "Goals of Intermediate Difficulty" (GOID). By training a generator to produce goals that the agent can solve some of the time, but not always, the system creates a self-regulating learning signal. Similarly, ALP-GMM acts as a "teacher" algorithm that tracks where a student is making the most "Absolute Learning Progress," focusing training on the frontier of the agent's capabilities.

Asymmetric Self-Play: Alice and Bob

One of the most elegant ways to generate a curriculum without any external reward engineering is through Asymmetric Self-Play. In this framework, introduced by Sainbayar Sukhbaatar and Arthur Szlam in Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play, an agent is split into two roles: Alice and Bob.

  1. Alice performs a sequence of actions to reach a state and then challenges Bob to either return to the start or reach that same state.
  2. If Bob fails, Alice has found a task that is "just right" for Bob to learn from.
  3. If Bob succeeds easily, Alice must find a more difficult challenge.

This "challenge and response" game creates a natural progression. OpenAI researchers later demonstrated in Asymmetric self-play for automatic goal discovery that this method allows robots to master complex manipulation tasks, like table setting, entirely through self-generated goals.

Morphing the Task Distribution

While some methods generate discrete goals, others view the curriculum as a continuous "morphing" of the environment. Pascal Klink and Jan Peters proposed Self-Paced Contextual Reinforcement Learning, which dynamically adjusts the distribution of tasks. It starts with a wide, easy distribution and slowly narrows it toward the difficult target task as the agent's performance improves.

Taking this further, Peide Huang and colleagues introduced GRADIENT, which uses Optimal Transport to find the smoothest path (a Wasserstein barycenter) between an easy source distribution and a hard target distribution. This ensures that the agent never faces a "difficulty spike" that could cause its policy to collapse.

Environment as Policy

In high-dimensional tasks like drone racing or autonomous driving, the "curriculum" isn't just a goal—it's the entire world layout. Hongze Wang and Davide Scaramuzza developed an Environment Policy that acts as an automated coach, procedurally designing race tracks that are just challenging enough for the agent.

Similarly, Ahmed Abouelazm showed in Automatic Curriculum Learning for Driving Scenarios that a teacher agent can mutate traffic density and weather conditions to keep the driving agent in the optimal learning zone, improving success rates by 21% over standard training methods.

Why It Works: Internalizing the Path

The ultimate power of ACL is that it allows the agent to "internalize" the structure of the problem. Benjamin Eysenbach and Sergey Levine argue in C-Planning that by using graph search to find intermediate waypoints during training, the agent learns a policy that inherently understands how to decompose long-horizon tasks.

By automating the discovery of these intermediate steps, researchers like John B. Lanier have achieved milestones once thought impossible for sparse-reward RL, such as stacking multiple blocks without a single human demonstration. ACL transforms the daunting "needle-in-a-haystack" search of sparse rewards into a guided ascent up a mountain of increasing complexity.

Papers in this version

Showing 60 of 151 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 87 papers
  2. Version 1 · 2026-10-04 · 151 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_manual_curriculum_design_5e39f153,
  title = {manual curriculum design},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/manual-curriculum-design/rdr_5e39f1533d754ff3958dd93cf5c61708},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_automated_curriculum_generation_for_spar_5e39f153_v1,
  title = {Automated Curriculum Generation for Sparse Reward Environments},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automated-curriculum-generation-for-sparse-reward-environments/rdr_5e39f1533d754ff3958dd93cf5c61708/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}