Learning to Execute: Efficient Learning of Universal Plan-Conditioned Policies in Robotics

Ingmar Schubert, Danny Driess, Ozgur S. Oguz, Marc Toussaint

Abstract

Reinforcement Learning (RL) provides a flexible framework for robotic control but is often hindered by extreme sample complexity in sparse-reward environments. Conversely, model-based planning is efficient but brittle to modeling errors. We present Learning to Execute (L2E), a framework that bridges this gap by conditioning RL policies on approximate, "crude" plans. L2E employs Final-Volume-Preserving Reward Shaping (FV-RS) to guide exploration along these plans without distorting the optimal policy. Our results in simulated manipulation tasks show that L2E achieves superior success rates and sample efficiency compared to state-of-the-art RL baselines, even when provided with highly simplified trajectories.

Introduction

Robotics research faces a persistent trade-off between the flexibility of Reinforcement Learning and the efficiency of model-based planning. While RL can learn complex behaviors from scratch, it often requires millions of interactions, which is impractical for real-world hardware. Model-based planners can generate paths quickly but fail when the underlying physics model—such as friction or contact dynamics—is inaccurate.

The core challenge in sparse-reward RL is exploration. Without a dense signal, an agent may never encounter the goal. This paper introduces L2E, which transforms the goal-reaching problem into a plan-following task. By using imperfect plans as a guide, L2E facilitates efficient exploration while maintaining the robustness to handle plan inaccuracies.

Methodology

L2E introduces three primary technical innovations:

  1. Plan-Conditioned Policies: Policies are conditioned on a sequence of states representing a plan, $\pi(a|s, p)$, rather than just a final goal.
  2. Final-Volume-Preserving Reward Shaping (FV-RS): This technique provides dense rewards for following the plan. Crucially, FV-RS is designed to preserve the optimal policy; if a plan is physically impossible or suboptimal, the agent is still incentivized to reach the ultimate goal.
  3. Plan Replay Strategies: Extending Hindsight Experience Replay (HER), L2E uses a reward-biased replay buffer. By sampling plans that would have yielded high rewards for past trajectories, the agent learns more effectively from its own successes and failures.

Experimental Setup

The framework was evaluated using the Nvidia PhysX engine across three environments:

  • Basic Pushing: A spherical end-effector pushes a box to a target.
  • Obstacle Pushing: A more complex version requiring navigation around a central barrier.
  • 2D Maze: A dynamic environment used to test zero-shot adaptation to changing configurations.
Basic pushing task visualization and success rate comparison.

Results and Discussion

L2E demonstrated significant performance gains over pure RL baselines like HER. In the basic pushing task, L2E variants reached a ~90% success rate with far fewer transitions than standard methods.

Obstacle pushing task visualization and success rate comparison.

A key finding is L2E's robustness to "crude" plans. In the obstacle avoidance task, the planner provided simple "Manhattan-style" paths. The L2E agent learned to smooth these into efficient, continuous motions, discovering contact-rich strategies that the planner could not model. Furthermore, in the 2D maze task, the plan-conditioned policy allowed for zero-shot adaptation to new environment layouts, as the agent could immediately follow a new plan generated by a simple RRT planner.

Conclusion

L2E provides a robust framework for combining the strengths of planning and learning. By treating plans as informative suggestions rather than strict commands, robots can learn to navigate complex environments with high sample efficiency. Future work will focus on addressing globally misleading plans and transitioning these capabilities from simulation to real-world robotic systems.

Summary updated

Abstract

Applications of Reinforcement Learning (RL) in robotics are often limited by high data demand. On the other hand, approximate models are readily available in many robotics scenarios, making model-based approaches like planning a data-efficient alternative. Still, the performance of these methods suffers if the model is imprecise or wrong. In this sense, the respective strengths and weaknesses of RL and model-based planners are complementary. In the present work, we investigate how both approaches can be integrated into one framework that combines their strengths. We introduce Learning to Execute (L2E), which leverages information contained in approximate plans to learn universal policies that are conditioned on plans. In our robotic manipulation experiments, L2E exhibits increased performance when compared to pure RL, pure planning, or baseline methods combining learning and planning.
BibTeX
@inproceedings{schubert2021learning,
  author = {Ingmar Schubert and Danny Driess and Ozgur S. Oguz and Marc Toussaint},
  title = {{Learning to Execute: Efficient Learning of Universal Plan-Conditioned Policies in Robotics}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=yiWW5YsEGIy},
}
Figure