Abstract
In Deep Reinforcement Learning (RL), sparse rewards often hinder agent training, particularly in tasks requiring high precision. This paper proposes Accuracy-based Curriculum Learning, a method that dynamically adjusts an agent's success threshold based on its learning progress. By utilizing an automated "competence progress" metric to select accuracy levels, the proposed ACTIVE-$ε$ strategy allows agents to build their own curricula. Experimental results on the OpenAI Gym Reacher environment demonstrate that this approach significantly outperforms fixed-accuracy baselines and random-accuracy training in both efficiency and stability.
Introduction
A primary hurdle in Deep RL is the "sparse reward" problem: if an agent only receives a reward for perfect task completion, it may never receive enough signal to learn. While curriculum learning—starting with easier versions of a task—can mitigate this, manually designing these sequences is often tedious and environment-specific. This research explores whether a universal curriculum can be created by simply varying the required precision for success. For a robotic reaching task, this means transitioning from a loose target (e.g., 10cm) to a high-precision one (e.g., 1mm) as the agent improves.
Methodology
The researchers implemented a Deep Deterministic Policy Gradient (DDPG) agent extended with Universal Value Function Approximators (UVFA).
Reward and State Representation
The agent receives a sparse reward of 0 if it is within a distance $ε$ of the target, and -1 otherwise. To enable the agent to understand varying precision levels, the accuracy requirement $ε$ is included directly in the input state. This allows the neural network to learn a multi-fidelity policy that generalizes across different precision requirements.
ACTIVE-$ε$ Curriculum
The core novelty is the ACTIVE-$ε$ engine, which tracks "competence progress" ($cp$) across a set of accuracy levels $E = \{0.02, 0.03, 0.04, 0.05\}$. Competence progress measures the rate of improvement in the agent's success rate at a specific $ε$. The system then samples accuracy levels using a power-law distribution: $P(ε_i) \propto cp_i^\beta$. This mechanism ensures the agent focuses on the "Goldilocks zone"—tasks that are neither too easy (already mastered) nor too hard (no progress yet).
Key Results and Discussion
Performance and Stability
The ACTIVE-$ε$ strategy demonstrated superior efficiency compared to baseline DDPG. As shown in Figure 1, the curriculum-driven approach reaches high accuracy levels much faster. Notably, the method significantly reduces variance between training runs; while baseline agents often fail to converge on certain seeds, ACTIVE-$ε$ agents are consistently successful.
Automatic Phasing
The data reveals a natural progression in the training process. Initially, the agent prioritizes the easiest accuracy level ($ε=0.05$). As competence at this level saturates, the system automatically shifts its focus toward tighter requirements.
By the end of the training period, the most difficult accuracy level ($ε=0.02$) becomes the primary focus. This "automatic phasing" confirms that the agent effectively constructs its own learning path without manual intervention.
Conclusion
Accuracy-based curriculum learning provides a robust framework for training RL agents on high-precision tasks. By leveraging UVFAs and a competence-based sampling strategy, agents can learn the underlying physics of an environment through easier successes before tackling high-precision goals. Future work includes comparing this explicit curriculum with implicit methods like Hindsight Experience Replay (HER) and testing the approach on higher-dimensional robotic tasks.
Summary updated