Automatic Curriculum Learning through Value Disagreement

Yunzhi Zhang, Pieter Abbeel, Lerrel Pinto

Training reinforcement learning (RL) agents to master multiple goals is notoriously sample-inefficient. When goals are sampled randomly, agents often spend their time failing at impossibly hard tasks or repeating tasks they have already mastered. This paper introduces Value Disagreement Sampling (VDS), a simple yet powerful method that automatically builds a training curriculum by prioritizing goals at the "frontier" of an agent's capabilities, significantly speeding up learning across complex robotic manipulation and navigation tasks.

The Challenge of Multi-Goal Reinforcement Learning

In multi-goal RL, an agent must learn a "universal policy" capable of reaching any given target state. However, in environments with sparse rewards—where the agent only receives a binary success or failure signal—learning is incredibly slow.

Humans do not learn by attempting random, disconnected tasks; we follow a curriculum. For instance, a baby learns to palm an object before mastering fine-finger manipulation. While researchers have attempted to automate this via "curriculum learning," previous approaches often rely on unstable adversarial setups (like a "teacher" agent trying to stump a "student") or complex, hand-engineered heuristics that fail to scale well to new environments.

Value Disagreement Sampling (VDS)

To overcome these limitations, the authors propose a Goal Proposal Module based on the concept of epistemic uncertainty—essentially, measuring what the model knows it doesn't know.

The Goal Proposal Module samples goals at the frontier between solvable and unsolvable regions.

Instead of training a single Q-function (which estimates the expected reward for an action), VDS trains an ensemble of $K$ independent Q-functions. For any potential goal, the system measures the standard deviation (disagreement) between the predictions of these Q-functions:

  • Too Easy: All Q-functions agree the reward will be high.
  • Too Hard: All Q-functions agree the reward will be low or zero.
  • The Frontier: At the edge of the agent's current ability, some Q-functions predict success while others predict failure. This high disagreement signals a goal that is "just right" for learning.

The agent then samples new training goals from a distribution weighted by this disagreement. The core insight is that the value function itself contains the map for the curriculum. You don't need a separate "teacher" agent; you just need to look at where your current "student" is most confused.

VDS is highly modular. In this study, it is paired with DDPG (a standard RL algorithm) and Hindsight Experience Replay (HER). While HER helps the agent learn from any data it happens to collect by pretending a reached state was the intended goal, VDS ensures the agent collects the most informative data in the first place.

Experimental Results

The researchers evaluated VDS on 18 challenging sparse-reward tasks, including robotic manipulation and complex navigation.

VDS was evaluated across 18 tasks, including Fetch robot manipulation, Shadow Hand manipulation, and Ant maze navigation.
  • 13 Robotic Manipulation Tasks: Drawn from the OpenAI Gym suite, these include Fetch (Reach, Push, Slide, Pick & Place) and Shadow Hand Manipulation (moving blocks, rotating eggs, or manipulating a pen).
  • 5 Navigation Tasks: Including 3 Maze Navigation tasks to visualize how the goal distribution expands over time, and 2 Ant Navigation tasks featuring complex quadruped robots navigating a maze.

Key Findings: * Superior Sample Efficiency: VDS outperformed standard HER and other state-of-the-art curriculum methods (like RIAC and ALP-GMM) on almost all benchmark environments. * Complex Manipulation: On highly complex tasks like "HandManipulateBlockRotateZ", VDS reached high success rates significantly faster than baselines, which often plateaued at much lower performance levels. * Outperforming Adversaries: In the Ant navigation tasks, VDS achieved better success rates with fewer samples than GoalGAN, despite GoalGAN being specifically designed for those environments. VDS avoids the "black art" of balancing two competing networks required by adversarial methods.

Limitations and Future Work

While VDS presents a significant step forward, it has a few limitations:

  • State-Based vs. Vision-Based: The experiments primarily used state-based observations (coordinates, joint angles). Extending VDS to visual reinforcement learning (pixels) remains future work.
  • Computational Overhead: Training an ensemble of Q-functions is more computationally expensive than training one, though the authors note their experiments still ran efficiently on a single CPU.
  • Safety: The curriculum is driven by uncertainty, which doesn't inherently account for safety. In real-world robotics, "frontier" goals might involve risky maneuvers that could damage hardware.
  • Exploration: VDS optimizes goal selection but doesn't solve the fundamental problem of exploration in environments where the agent has never seen a reward signal at all. It assumes the agent can at least occasionally stumble upon a reward to kickstart the ensemble's disagreement.

Summary updated

Abstract

Continually solving new, unsolved tasks is the key to learning diverse behaviors. Through reinforcement learning (RL), we have made massive strides towards solving tasks that have a single goal. However, in the multi-task domain, where an agent needs to reach multiple goals, the choice of training goals can largely affect sample efficiency. When biological agents learn, there is often an organized and meaningful order to which learning happens. Inspired by this, we propose setting up an automatic curriculum for goals that the agent needs to solve. Our key insight is that if we can sample goals at the frontier of the set of goals that an agent is able to reach, it will provide a significantly stronger learning signal compared to randomly sampled goals. To operationalize this idea, we introduce a goal proposal module that prioritizes goals that maximize the epistemic uncertainty of the Q-function of the policy. This simple technique samples goals that are neither too hard nor too easy for the agent to solve, hence enabling continual improvement. We evaluate our method across 13 multi-goal robotic tasks and 5 navigation tasks, and demonstrate performance gains over current state-of-the-art methods.
BibTeX
@inproceedings{zhang2020automatic,
  author = {Yunzhi Zhang and Pieter Abbeel and Lerrel Pinto},
  title = {{Automatic Curriculum Learning through Value Disagreement}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=wsVKcUv-Qpm},
}
Figure