Automatic Curriculum Learning through Value Disagreement

Yunzhi Zhang, Pieter Abbeel, Lerrel Pinto

Abstract

Continually solving new, unsolved tasks is the key to learning diverse behaviors. Through reinforcement learning (RL), we have made massive strides towards solving tasks that have a single goal. However, in the multi-task domain, where an agent needs to reach multiple goals, the choice of training goals can largely affect sample efficiency. When biological agents learn, there is often an organized and meaningful order to which learning happens. Inspired by this, we propose setting up an automatic curriculum for goals that the agent needs to solve. Our key insight is that if we can sample goals at the frontier of the set of goals that an agent is able to reach, it will provide a significantly stronger learning signal compared to randomly sampled goals. To operationalize this idea, we introduce a goal proposal module that prioritizes goals that maximize the epistemic uncertainty of the Q-function of the policy. This simple technique samples goals that are neither too hard nor too easy for the agent to solve, hence enabling continual improvement. We evaluate our method across 13 multi-goal robotic tasks and 5 navigation tasks, and demonstrate performance gains over current state-of-the-art methods.

Introduction

Reinforcement Learning (RL) has achieved remarkable success in controlled environments, yet it remains significantly sample inefficient. Learning a single complex skill often requires millions of trials. This challenge is exacerbated in multi-task or goal-conditioned RL, where an agent must master a variety of objectives. Randomly selecting training goals often leads to inefficient learning: agents either focus on "too easy" goals they have already mastered or "too hard" goals that are currently unreachable, providing no useful reward signal.

Drawing inspiration from biological learning, where tasks are mastered in a structured sequence, we propose an automated curriculum learning approach. The goal is to identify and prioritize tasks at the "Goldilocks" zone—the frontier of the agent's current capabilities.

Methods: Value Disagreement based Sampling (VDS)

The core of our approach is Value Disagreement based Sampling (VDS). We identify the learning frontier by measuring the epistemic uncertainty of the agent's internal model.

Conceptual overview of VDS targeting the frontier between solvable and unsolvable goals.

The Ensemble and Disagreement Metric

Instead of a single Q-function, VDS employs an ensemble of $K$ Q-functions (typically $K=3$) with randomized initializations. The disagreement among these functions serves as a proxy for the learning frontier:

  1. Low Disagreement (High Success): If all Q-functions predict a high probability of success, the goal is likely too easy.
  2. Low Disagreement (Low Success): If all Q-functions predict failure, the goal is likely too difficult or unreachable.
  3. High Disagreement: When predictions vary significantly, the goal lies at the edge of the agent's knowledge, offering the highest potential for learning.

Goal Proposal and Integration

The system samples potential goals and weights them based on the variance (disagreement) of the ensemble's predictions. VDS is modular and was evaluated in combination with Hindsight Experience Replay (HER) and DDPG, though it is compatible with other off-policy RL algorithms.

Experimental Evaluation

We evaluated VDS across 18 challenging sparse-reward environments, including 13 robotic manipulation tasks (Fetch and Hand environments) and 5 navigation tasks (Maze and Ant-embodied agents).

Success rate comparison across 16 tasks, showing VDS+HER (red) consistently outperforming baselines.

Key Results

  • Superior Sample Efficiency: VDS significantly reduced the number of timesteps required to reach high success rates. In complex "Hand" manipulation tasks, VDS often achieved in half the time what vanilla HER required.
  • Robustness to Task Complexity: On high-dimensional tasks like HandManipulateBlockRotateXYZ, VDS was frequently the only method to achieve meaningful progress within the training budget.
  • Scalability: In navigation benchmarks, VDS outperformed prior curriculum methods like GoalGAN, demonstrating better scalability to high-dimensional state spaces.
  • Complementarity with HER: While HER allows learning from unintended successes, VDS proactively selects which goals to pursue. The combination of both techniques yielded the strongest results.

Conclusion

Value Disagreement based Sampling provides a simple yet effective mechanism for automatic curriculum generation in RL. By leveraging ensemble uncertainty to target the learning frontier, VDS avoids the instabilities of adversarial training while significantly improving sample efficiency across diverse robotic and navigation tasks. Future work will explore applying VDS to visual reinforcement learning and physical robotic hardware.

Summary updated

Abstract

Continually solving new, unsolved tasks is the key to learning diverse behaviors. Through reinforcement learning (RL), we have made massive strides towards solving tasks that have a single goal. However, in the multi-task domain, where an agent needs to reach multiple goals, the choice of training goals can largely affect sample efficiency. When biological agents learn, there is often an organized and meaningful order to which learning happens. Inspired by this, we propose setting up an automatic curriculum for goals that the agent needs to solve. Our key insight is that if we can sample goals at the frontier of the set of goals that an agent is able to reach, it will provide a significantly stronger learning signal compared to randomly sampled goals. To operationalize this idea, we introduce a goal proposal module that prioritizes goals that maximize the epistemic uncertainty of the Q-function of the policy. This simple technique samples goals that are neither too hard nor too easy for the agent to solve, hence enabling continual improvement. We evaluate our method across 13 multi-goal robotic tasks and 5 navigation tasks, and demonstrate performance gains over current state-of-the-art methods.
BibTeX
@inproceedings{zhang2020automatic,
  author = {Yunzhi Zhang and Pieter Abbeel and Lerrel Pinto},
  title = {{Automatic Curriculum Learning through Value Disagreement}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=wsVKcUv-Qpm},
}
Figure