Accuracy-based Curriculum Learning in Deep Reinforcement Learning

Pierre Fournier, Olivier Sigaud, Mohamed Chetouani, Pierre-Yves OudeyerarXiv 2018 · arXiv

Abstract

In Deep Reinforcement Learning (RL), sparse rewards often hinder agent training, particularly in tasks requiring high precision. This paper proposes Accuracy-based Curriculum Learning, a method that dynamically adjusts an agent's success threshold based on its learning progress. By utilizing an automated "competence progress" metric to select accuracy levels, the proposed ACTIVE-$ε$ strategy allows agents to build their own curricula. Experimental results on the OpenAI Gym Reacher environment demonstrate that this approach significantly outperforms fixed-accuracy baselines and random-accuracy training in both efficiency and stability.

Introduction

A primary hurdle in Deep RL is the "sparse reward" problem: if an agent only receives a reward for perfect task completion, it may never receive enough signal to learn. While curriculum learning—starting with easier versions of a task—can mitigate this, manually designing these sequences is often tedious and environment-specific. This research explores whether a universal curriculum can be created by simply varying the required precision for success. For a robotic reaching task, this means transitioning from a loose target (e.g., 10cm) to a high-precision one (e.g., 1mm) as the agent improves.

Methodology

The researchers implemented a Deep Deterministic Policy Gradient (DDPG) agent extended with Universal Value Function Approximators (UVFA).

Reward and State Representation

The agent receives a sparse reward of 0 if it is within a distance $ε$ of the target, and -1 otherwise. To enable the agent to understand varying precision levels, the accuracy requirement $ε$ is included directly in the input state. This allows the neural network to learn a multi-fidelity policy that generalizes across different precision requirements.

ACTIVE-$ε$ Curriculum

The core novelty is the ACTIVE-$ε$ engine, which tracks "competence progress" ($cp$) across a set of accuracy levels $E = \{0.02, 0.03, 0.04, 0.05\}$. Competence progress measures the rate of improvement in the agent's success rate at a specific $ε$. The system then samples accuracy levels using a power-law distribution: $P(ε_i) \propto cp_i^\beta$. This mechanism ensures the agent focuses on the "Goldilocks zone"—tasks that are neither too easy (already mastered) nor too hard (no progress yet).

Key Results and Discussion

Performance and Stability

The ACTIVE-$ε$ strategy demonstrated superior efficiency compared to baseline DDPG. As shown in Figure 1, the curriculum-driven approach reaches high accuracy levels much faster. Notably, the method significantly reduces variance between training runs; while baseline agents often fail to converge on certain seeds, ACTIVE-$ε$ agents are consistently successful.

Figure 1: Comparison of agent accuracy over training steps for Baseline DDPG, RANDOM-ε, and ACTIVE-ε. The curriculum approach shows faster learning and significantly lower variance.

Automatic Phasing

The data reveals a natural progression in the training process. Initially, the agent prioritizes the easiest accuracy level ($ε=0.05$). As competence at this level saturates, the system automatically shifts its focus toward tighter requirements.

Figure 2: The proportion of different accuracy levels (ε) sampled over time. The agent naturally transitions from easier (higher ε) to harder (lower ε) tasks as it learns.

By the end of the training period, the most difficult accuracy level ($ε=0.02$) becomes the primary focus. This "automatic phasing" confirms that the agent effectively constructs its own learning path without manual intervention.

Conclusion

Accuracy-based curriculum learning provides a robust framework for training RL agents on high-precision tasks. By leveraging UVFAs and a competence-based sampling strategy, agents can learn the underlying physics of an environment through easier successes before tackling high-precision goals. Future work includes comparing this explicit curriculum with implicit methods like Hindsight Experience Replay (HER) and testing the approach on higher-dimensional robotic tasks.

Summary updated

Abstract

In this paper, we investigate a new form of automated curriculum learning based on adaptive selection of accuracy requirements, called accuracy-based curriculum learning. Using a reinforcement learning agent based on the Deep Deterministic Policy Gradient algorithm and addressing the Reacher environment, we first show that an agent trained with various accuracy requirements sampled randomly learns more efficiently than when asked to be very accurate at all times. Then we show that adaptive selection of accuracy requirements, based on a local measure of competence progress, automatically generates a curriculum where difficulty progressively increases, resulting in a better learning efficiency than sampling randomly.
BibTeX
@misc{fournier2018accuracy,
  author = {Pierre Fournier and Olivier Sigaud and Mohamed Chetouani and Pierre-Yves Oudeyer},
  title = {{Accuracy-based Curriculum Learning in Deep Reinforcement Learning}},
  year = {2018},
  eprint = {1806.09614},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1806.09614},
}
Figure