Model Learning for Look-ahead Exploration in Continuous Control

Arpit Agarwal, Katharina Muelling, Katerina Fragkiadaki

Abstract

We propose an exploration method that incorporates look-ahead search over basic learnt skills and their dynamics, and use it for reinforcement learning (RL) of manipulation policies. Our skills are multi-goal policies learned in isolation in simpler environments using existing multigoal RL formulations, analogous to options or macroactions. Coarse skill dynamics, i.e., the state transition caused by a (complete) skill execution, are learnt and are unrolled forward during lookahead search. Policy search benefits from temporal abstraction during exploration, though itself operates over low-level primitive actions, and thus the resulting policies does not suffer from suboptimality and inflexibility caused by coarse skill chaining. We show that the proposed exploration strategy results in effective learning of complex manipulation policies faster than current state-of-the-art RL methods, and converges to better policies than methods that use options or parametrized skills as building blocks of the policy itself, as opposed to guiding exploration.

Introduction

Reinforcement Learning (RL) has shown great promise in robotic manipulation, yet it frequently struggles with the challenge of sparse rewards. In tasks like block stacking or container placement, an agent may receive no feedback for millions of steps until a success is accidentally achieved. Traditional Hierarchical RL (HRL) attempts to solve this by using high-level "macro-actions" or options. However, this often leads to a trade-off: using coarse macro-actions can accelerate learning but results in "clunky," suboptimal final policies that are limited by the quality of the pre-defined skills.

This paper introduces a "best of both worlds" approach: using high-level, learned skills to "imagine" successful paths during exploration, while maintaining a fine-grained, low-level policy for execution. By performing look-ahead tree searches over coarse skill dynamics, the agent can solve complex tasks much faster without sacrificing the precision of low-level control.

The HERLASE Framework

The proposed method, HERLASE (Hindsight Experience Replay with Look-Ahead Search Exploration), decouples the exploration mechanism from the final control policy. The framework consists of four primary components:

  1. Skill Acquisition: The agent first learns a library of basic skills (e.g., reach, grasp, transfer) in simplified environments using Hindsight Experience Replay (HER).
  2. Learning Coarse Dynamics: For each skill, the agent learns a transition function—a mental model predicting the state resulting from executing a skill toward a specific sub-goal. It also estimates the success probability of these transitions.
  3. Look-ahead Exploration: During training for a complex task, the agent performs a tree search, "mentally unrolling" sequences of learned skills to identify paths leading to the final goal. This is controlled by a parameter $z$, where a search is performed if $z < \epsilon$.
  4. Low-level Execution: Crucially, while exploration is guided by high-level skills, the policy being trained operates on low-level motor commands (task space coordinates). This ensures the final behavior is smooth and not constrained by the coarse nature of the skills.
Overview of the HERLASE framework, illustrating the interaction between the skill library, look-ahead search, and the low-level policy.

The core of the exploration strategy is the look-ahead tree. By unrolling the learned coarse dynamics, the agent can evaluate potential future states (mental states) and their associated rewards without physical interaction. This "purposeful exploration" allows the agent to actively hunt for sequences that lead to rewards, rather than relying solely on random noise or curiosity-based exploration.

Detailed view of the look-ahead tree search unrolling mental states to find the maximum utility path.

Experimental Evaluation

The researchers evaluated HERLASE in the MuJoCo physics simulator using a 7-degree-of-freedom Baxter robot arm. They tested the method on four increasingly difficult manipulation tasks:

  • Pick and Move: Grasping an object and moving it to a 3D target.
  • Put A inside B: Placing an object inside a container.
  • Stack A on top of B: Stacking one object on another.
  • Take A out of B: Removing an object from a container.

These tasks are characterized by long horizons and sparse rewards, making them difficult for standard RL algorithms.

Key Results and Insights

HERLASE significantly outperformed standard HER and other hierarchical baselines. For instance, in the "Put A inside B" task, standard HER failed to learn (0% success), while HERLASE achieved near-perfect success.

Several key insights emerged from this work:

  • Coarse Dynamics Robustness: Predicting the outcome of a multi-step skill (like a "grasp") is often more robust than predicting the outcome of many individual low-level motor increments, as errors accumulate more slowly.
  • Decoupling Search from Representation: By using skills only to populate the replay buffer with successful trajectories, the agent learns from high-quality data without being forced to use those specific skills in the final execution.
  • Sample Efficiency: The method reaches high success rates significantly faster than methods without look-ahead search.

Limitations and Future Work

While effective, HERLASE faces a computational bottleneck due to the "thinking time" required for tree searches. A high branching factor can significantly increase the time per episode. Additionally, the current implementation uses static skill models and relies on state coordinates (centroids) rather than raw visual input. Future work could involve updating skill models online and extending the framework to operate directly on pixel data.

Summary updated

Abstract

We propose an exploration method that incorporates look-ahead search over basic learnt skills and their dynamics, and use it for reinforcement learning (RL) of manipulation policies . Our skills are multi-goal policies learned in isolation in simpler environments using existing multigoal RL formulations, analogous to options or macroactions. Coarse skill dynamics, i.e., the state transition caused by a (complete) skill execution, are learnt and are unrolled forward during lookahead search. Policy search benefits from temporal abstraction during exploration, though itself operates over low-level primitive actions, and thus the resulting policies does not suffer from suboptimality and inflexibility caused by coarse skill chaining. We show that the proposed exploration strategy results in effective learning of complex manipulation policies faster than current state-of-the-art RL methods, and converges to better policies than methods that use options or parametrized skills as building blocks of the policy itself, as opposed to guiding exploration. We show that the proposed exploration strategy results in effective learning of complex manipulation policies faster than current state-of-the-art RL methods, and converges to better policies than methods that use options or parameterized skills as building blocks of the policy itself, as opposed to guiding exploration.
BibTeX
@inproceedings{agarwal2019model,
  author = {Arpit Agarwal and Katharina Muelling and Katerina Fragkiadaki},
  title = {{Model Learning for Look-ahead Exploration in Continuous Control}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=keXhNHKCIqP},
}
Figure