Curriculum Reinforcement Learning via Constrained Optimal Transport

Pascal Klink, Haoyi Yang, Carlo D'Eramo, Jan Peters, Joni PajarinenICML 2022 · OpenReview

The Challenge of Sparse Rewards

Reinforcement Learning (RL) agents often struggle with "sparse reward" environments—tasks where the agent receives no feedback until it perfectly completes a complex goal. Curriculum Reinforcement Learning (CRL) addresses this by starting the agent on easy tasks and gradually increasing the difficulty.

A popular way to automate this process is to treat the curriculum as an interpolation between an "easy" distribution of tasks and a "target" distribution. However, most existing methods use Kullback-Leibler (KL) divergence to measure the similarity between these distributions. The authors of this paper identify a major flaw in this approach: KL divergence is "metric-blind."

Geometric interpolation (related to KL divergence) fails to move probability mass across the task space, instead just fading modes in and out.

Because KL divergence only cares about the overlap in probability—and not how far apart two tasks are in the physical parameter space (e.g., a 1-meter jump versus a 5-meter jump)—it often leads to curricula that skip crucial intermediate "stepping stone" tasks. It can also fail entirely when the target tasks are multi-modal or non-Gaussian.

Introducing CURROT: Curricula via Optimal Transport

To solve this, the paper introduces CURROT (Curricula via Optimal Transport), a novel framework that replaces traditional KL-divergence-based task interpolation with Constrained Optimal Transport.

Instead of KL divergence, CURROT uses the 2-Wasserstein distance ($W_2$), which explicitly accounts for the geometric distance between task parameters. The core of CURROT is a constrained optimization problem designed to safely guide the agent:

  1. Minimize the Wasserstein distance between the current training distribution and the target distribution.
  2. Subject to a performance constraint: Ensure that the tasks being sampled are ones the agent can actually solve (achieving a return above a specific threshold $\delta$).
  3. Subject to a step-size constraint: Limit how much the distribution can change in one update to prevent the agent from being overwhelmed.
The effect of the performance constraint $\delta$ on the curriculum progression, showing how the task distribution shifts while respecting the agent's competence boundary.

To make this computationally feasible, CURROT represents task distributions as sets of particles. Updating the curriculum involves solving a linear assignment problem (a classic Optimal Transport problem) to move these particles toward the target while staying strictly within the agent's "boundary of competence."

Key Insights and Results

The authors evaluated CURROT on three distinct benchmarks designed to highlight different CRL challenges: Sparse Goal Reaching (SGR), Point Mass, and Bipedal Walker Stump Tracks.

  • The Geometry Matters: Because Optimal Transport considers the metric of the task space, CURROT "pushes" probability mass along physical paths. If an agent needs to learn to jump 1m before 2m, CURROT’s use of Wasserstein distance naturally prioritizes that progression.
  • Breaking the Gaussian Constraint: Many previous methods assume the task distribution is a single Gaussian "blob." CURROT’s particle-based approach allows it to split, stretch, and wrap around obstacles. In the Point Mass experiment, CURROT successfully learned to navigate a bi-modal target (two possible gates), whereas Gaussian-based methods could only focus on one gate, effectively ignoring half of the target distribution.
  • Focusing on the "Boundary": The performance constraint implicitly forces the curriculum to stay at the edge of what the agent knows, preventing the "forgetting" or "stagnation" seen in methods that try to cover the whole task space uniformly.
  • Superior Performance and Robustness: In the Sparse Goal Reaching task, CURROT outperformed all other CRL baselines, coming closest to an "Oracle" that had perfect knowledge of the feasible task space. Furthermore, in a "Mostly Infeasible" Bipedal Walker setting, CURROT was statistically significantly better than all other methods, proving it is highly efficient at finding the narrow "path of learnability" in a sea of impossible tasks.

Limitations and Future Work

While highly effective, CURROT does have limitations. The cost of Optimal Transport scales cubically ($O(n^3)$) with the number of particles, which might limit its use for extremely high-dimensional task spaces. Additionally, the method relies on hyperparameter tuning for the performance threshold ($\delta$) and distance bound ($\epsilon$), requiring some environment-specific knowledge.

Currently, the work assumes a Euclidean task space. The authors suggest that exploring Wasserstein distances on manifolds or non-Euclidean graphs could be a promising future direction for applying CURROT to even more complex RL domains.

Summary updated

Abstract

Curriculum reinforcement learning (CRL) allows solving complex tasks by generating a tailored sequence of learning tasks, starting from easy ones and subsequently increasing their difficulty. Although the potential of curricula in RL has been clearly shown in a variety of works, it is less clear how to generate them for a given learning environment, resulting in a variety of methods aiming to automate this task. In this work, we focus on the idea of framing curricula as interpolations between task distributions, which has previously been shown to be a viable approach to CRL. Identifying key issues of existing methods, we frame the generation of a curriculum as a constrained optimal transport problem between task distributions. Benchmarks show that this way of curriculum generation can improve upon existing CRL methods, yielding high performance in a variety of tasks with different characteristics.
BibTeX
@inproceedings{klink2022curriculum,
  author = {Pascal Klink and Haoyi Yang and Carlo D'Eramo and Jan Peters and Joni Pajarinen},
  title = {{Curriculum Reinforcement Learning via Constrained Optimal Transport}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=6mGql9tOzY},
}
Figure