Model Learning for Look-Ahead Exploration in Continuous Control

Arpit Agarwal, Katharina Muelling, Katerina Fragkiadaki

Abstract

We propose an exploration method that incorporates lookahead search over basic learnt skills and their dynamics, and use it for reinforcement learning (RL) of manipulation policies. Our skills are multi-goal policies learned in isolation in simpler environments using existing multigoal RL formulations, analogous to options or macroactions. Coarse skill dynamics—the state transition caused by a complete skill execution—are learnt and unrolled forward during lookahead search. Policy search benefits from temporal abstraction during exploration, though itself operates over low-level primitive actions; thus, the resulting policies do not suffer from suboptimality and inflexibility caused by coarse skill chaining. We show that the proposed exploration strategy results in effective learning of complex manipulation policies faster than current state-of-the-art RL methods and converges to better policies than methods that use options or parameterized skills as building blocks of the policy itself.

Introduction

In robotic manipulation, agents frequently encounter "sparse rewards," where a positive signal is only received after completing a complex sequence of actions, such as reaching, grasping, and placing an object. Standard exploration strategies, like $\epsilon$-greedy noise, often fail in high-dimensional continuous spaces because random joint movements are unlikely to stumble upon these successful sequences.

While Hierarchical Reinforcement Learning (HRL) addresses this by using "options" or "macro-actions," it often introduces a trade-off between flexibility and learning speed. Forcing an agent to use coarse skills can lead to clunky, sub-optimal behavior. Conversely, relying solely on low-level primitives makes exploration prohibitively slow. This paper introduces HERLASE (Hindsight Experience Replay with Look-Ahead Search Exploration) to bridge this gap, using hierarchical efficiency for exploration without sacrificing the optimality of low-level control.

The HERLASE Framework

HERLASE decouples the exploration mechanism from the final policy representation. The framework operates in three distinct stages:

  1. Learning Basic Skills: The agent first learns a set of generalized skills (e.g., Reach, Grasp, Transfer) using Deep Deterministic Policy Gradient (DDPG) and Hindsight Experience Replay (HER). These are multi-goal policies capable of executing a skill toward varying targets.
  2. Learning Coarse Dynamics: For each skill, a neural network is trained to predict the outcome of its execution: given a starting state and a skill-goal, the model predicts the resulting state and the probability of success.
  3. Look-Ahead Exploration: During the training of a complex target task, the agent uses the learned dynamics to "mentally" unfold a search tree. It samples skill/sub-goal combinations to identify a path leading toward the final goal.
HERLASE System Architecture: The process integrates a skill library and learned dynamics into a look-ahead tree search to guide low-level exploration.

Crucially, while the search is hierarchical, the execution is not. The low-level transitions generated during these guided explorations are stored in a replay buffer. A "flat" DDPG+HER agent is then trained on this raw data, allowing it to learn a smooth, optimal policy that can eventually outperform the coarse skills used to find the reward initially.

Experimental Evaluation

The method was evaluated in the MuJoCo physics simulator using a 7-degree-of-freedom Baxter robot arm. The tasks included complex manipulation scenarios such as "Pick and Move," "Stack A on top of B," and "Put A inside B."

The Baxter robot arm in a simulated environment performing a 'Put A inside B' task, one of the complex manipulation scenarios used for evaluation.

Key Results

  • Success Rates: In the challenging "Put A inside B" task, standard HER (a state-of-the-art baseline) failed to achieve any success (0%), whereas HERLASE reached near 100% success.
  • Sample Efficiency: HERLASE reached high success rates significantly faster than hierarchical baselines like Parameterized Action Space (PAS), requiring fewer training epochs to solve complex tasks.
  • Optimality: Because the final learner optimizes low-level transitions rather than just chaining skills, HERLASE converged to more efficient and fluid policies than traditional HRL methods.

Discussion and Future Work

A core insight of this work is that purposeful dynamics are easier to learn than general world models. By focusing on the outcomes of goal-directed behaviors, the model-based component remains tractable. Furthermore, the system proved robust; even with imperfect dynamics models, the look-ahead search provided better guidance than random exploration.

Future research will focus on moving from coordinate-based state representations to learning directly from visual features (pixels). Additionally, updating the coarse dynamics models online during task training could further improve performance in dynamic or changing environments.

Summary updated

Abstract

We propose an exploration method that incorporates lookahead search over basic learnt skills and their dynamics, and use it for reinforcement learning (RL) of manipulation policies. Our skills are multi-goal policies learned in isolation in simpler environments using existing multigoal RL formulations, analogous to options or macroactions. Coarse skill dynamics, i.e., the state transition caused by a (complete) skill execution, are learnt and are unrolled forward during lookahead search. Policy search benefits from temporal abstraction during exploration, though itself operates over low-level primitive actions, and thus the resulting policies does not suffer from suboptimality and inflexibility caused by coarse skill chaining. We show that the proposed exploration strategy results in effective learning of complex manipulation policies faster than current state-of-the-art RL methods, and converges to better policies than methods that use options or parametrized skills as building blocks of the policy itself, as opposed to guiding exploration. We show that the proposed exploration strategy results in effective learning of complex manipulation policies faster than current state-of-the-art RL methods, and converges to better policies than methods that use options or parameterized skills as building blocks of the policy itself, as opposed to guiding exploration.
BibTeX
@inproceedings{agarwal2019model,
  author = {Arpit Agarwal and Katharina Muelling and Katerina Fragkiadaki},
  title = {{Model Learning for Look-Ahead Exploration in Continuous Control}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=keXhNHKCIqP},
}
Figure