Abstract Demonstrations and Adaptive Exploration for Efficient and Stable Multi-step Sparse Reward Reinforcement Learning

Xintong Yang, Ze Ji, Jing Wu, Yu-Kun Lai

Introduction

Reinforcement Learning (RL) faces a significant challenge in complex tasks where rewards are sparse and horizons are long. In robotic assembly, for instance, an agent may execute thousands of actions before receiving a success signal. Random exploration in such vast state spaces typically fails to converge. Researchers at Cardiff University have introduced the $A^2$ (Abstract Demonstrations and Adaptive Exploration) framework to address this "needle in a haystack" problem by mimicking human pedagogical strategies: high-level instruction followed by performance-based practice.

Problem Statement

Standard Deep Reinforcement Learning (DRL) algorithms struggle with multi-step tasks where the reward is only provided upon final completion. While kinematic demonstrations (recording precise joint movements) can guide agents, they are difficult to collect and often lack generalization. Conversely, standard exploration decay is task-agnostic, often leading to "forgetting" early subtasks before later ones are mastered.

The $A^2$ Framework

The $A^2$ framework decomposes learning into two synergistic components:

  1. Abstract Demonstrations (AD): Instead of precise trajectories, the agent is provided with a sequence of high-level subgoals. Utilizing Goal-conditioned RL and Hindsight Experience Replay (HER), the agent learns to navigate these milestones. This approach provides a strategic "instruction manual" while allowing the agent to discover its own optimal motor control.
Decomposition of the ChestPush task into abstract subgoals: Start, Open door, Reach block, and Push into chest.
  1. Adaptive Exploration (AE): This component dynamically adjusts exploration noise based on the agent's mastery of specific subtasks. If an agent consistently succeeds at a subtask, it reduces noise to act more deterministically. If success rates drop, stochasticity is increased to encourage further exploration. This ensures stability across the entire task chain.

Experimental Setup

The framework was evaluated in two distinct environments:

  • Mini-grid: A 2D grid-world requiring the agent to find keys and unlock doors.
  • Pybullet Multigoal (PMG): A 3D physics simulator for robotic manipulation tasks such as ChestPush, ChestPick, and BlockStack.
The ChestPush environment, where a robotic arm must open a container and manipulate a block.

Key Results and Insights

The integration of $A^2$ into off-policy algorithms like SAC, DDPG, and DQN yielded significant improvements:

  • Performance Gains: In the BlockStack task, the $A^2$-enhanced SAC agent achieved a ~90% success rate, compared to barely 20% for the vanilla version.
  • Optimal Demonstration Rate: The researchers found that a 75% demonstration rate is optimal; providing demonstrations in every episode prevents the agent from learning from failure, which is crucial for mapping state values accurately.
  • Stability and Variance: Adaptive Exploration significantly reduced variance between training runs, leading to more predictable and reliable convergence compared to task-agnostic exploration strategies.

Conclusion

The $A^2$ framework demonstrates that abstracting "what" to do while adaptively refining "how" to do it significantly enhances the efficiency of robot learning in sparse-reward environments. Future work aims to automate subtask discovery and transition to visual, image-based subgoals for broader real-world applicability.

Summary updated

Abstract

Although Deep Reinforcement Learning (DRL) has been popular in many disciplines including robotics, state-of-the-art DRL algorithms still struggle to learn long-horizon, multistep and sparse reward tasks, such as stacking several blocks given only a task-completion reward signal. To improve learning efficiency for such tasks, this paper proposes a DRL exploration technique, termed $\mathbf{A^{2}}$, which integrates two components inspired by human experiences: Abstract demonstrations and Adaptive exploration. $\mathbf{A^{2}}$ starts by decomposing a complex task into subtasks, and then provides the correct orders of subtasks to learn. During training, the agent explores the environment adaptively, acting more deterministically for well-mastered subtasks and more stochastically for ill-learnt subtasks. Ablation and comparative experiments are conducted on several grid-world tasks and three robotic manipulation tasks. We demonstrate that $\mathbf{A^{2}}$ can aid popular DRL algorithms (DQN, DDPG, and SAC) to learn more efficiently and stably in these environments.
BibTeX
@inproceedings{yang2022abstract,
  author = {Xintong Yang and Ze Ji and Jing Wu and Yu-Kun Lai},
  title = {{Abstract Demonstrations and Adaptive Exploration for Efficient and Stable Multi-step Sparse Reward Reinforcement Learning}},
  booktitle = {ICAC},
  year = {2022},
  url = {https://openreview.net/forum?id=OJdTMFeYxd-},
}

Loading related content...

Figure