Abstract
While recent advances in deep reinforcement learning have achieved impressive results in learning motor skills, many policies are only capable within a limited set of initial states. We propose an algorithm that automatically decomposes a hard task into a sequence of simpler subtasks by training a chain of local policies, termed "relay neural networks," which gradually and robustly expand a robot's motor skill repertoire.
Introduction
Standard Deep Reinforcement Learning (RL) often produces policies that succeed only from a narrow set of starting conditions. For instance, training a single monolithic policy to enable a robot to recover from a fall and then walk is computationally expensive and sample-inefficient. While Hierarchical RL addresses this by decomposing tasks, manual definition of subtasks requires significant domain expertise. This paper introduces a method to automatically discover these subtasks and smoothly chain their respective control policies.
Methodology
The proposed relay network is a directed graph of local policies trained via a reverse-curriculum approach. The process begins with a root policy ($\pi_0$) trained on an easy version of the task. A value function ($V_0$) is learned to estimate expected returns.
Crucially, the algorithm uses a decision tree classifier on the value function to define a "good zone" (states where $V_0(s) > \bar{V}_0$). Subsequent policies are trained to drive the robot into this zone, receiving a "relay reward" equal to the parent's value function at the handoff state. This explicit handoff mechanism allows for robust skill expansion.
Experimental Results
The method was evaluated on simulated continuous control tasks in the DART physics engine, including Cartpole swing-up, Hopper recovery from a supine position, and 2D Walker recovery from severe pushes.
The automated relay network (AUTO) consistently achieved higher average returns and better sample efficiency than training a single large policy (ONE) or using standard curriculum learning (CL). Ablation studies showed that the "relay reward" and terminal handoff conditions are vital; the "No Relay" (NR) baseline performed significantly worse.
Discussion and Conclusion
Value functions serve as effective boundary definers for subtasks, enabling the discovery of multi-modal skills. By identifying overlapping subtasks, the algorithm can merge policy chains, saving tens of thousands of training samples. However, the approach is sensitive to value function accuracy, which can limit the curriculum's step size. Future research will explore the theoretical bounds of optimality for these locally optimal chained policies to better understand the trade-off between local chaining and global performance.
Summary updated