Abstract
Reinforcement Learning (RL) provides a flexible framework for robotic control but is often hindered by extreme sample complexity in sparse-reward environments. Conversely, model-based planning is efficient but brittle to modeling errors. We present Learning to Execute (L2E), a framework that bridges this gap by conditioning RL policies on approximate, "crude" plans. L2E employs Final-Volume-Preserving Reward Shaping (FV-RS) to guide exploration along these plans without distorting the optimal policy. Our results in simulated manipulation tasks show that L2E achieves superior success rates and sample efficiency compared to state-of-the-art RL baselines, even when provided with highly simplified trajectories.
Introduction
Robotics research faces a persistent trade-off between the flexibility of Reinforcement Learning and the efficiency of model-based planning. While RL can learn complex behaviors from scratch, it often requires millions of interactions, which is impractical for real-world hardware. Model-based planners can generate paths quickly but fail when the underlying physics model—such as friction or contact dynamics—is inaccurate.
The core challenge in sparse-reward RL is exploration. Without a dense signal, an agent may never encounter the goal. This paper introduces L2E, which transforms the goal-reaching problem into a plan-following task. By using imperfect plans as a guide, L2E facilitates efficient exploration while maintaining the robustness to handle plan inaccuracies.
Methodology
L2E introduces three primary technical innovations:
- Plan-Conditioned Policies: Policies are conditioned on a sequence of states representing a plan, $\pi(a|s, p)$, rather than just a final goal.
- Final-Volume-Preserving Reward Shaping (FV-RS): This technique provides dense rewards for following the plan. Crucially, FV-RS is designed to preserve the optimal policy; if a plan is physically impossible or suboptimal, the agent is still incentivized to reach the ultimate goal.
- Plan Replay Strategies: Extending Hindsight Experience Replay (HER), L2E uses a reward-biased replay buffer. By sampling plans that would have yielded high rewards for past trajectories, the agent learns more effectively from its own successes and failures.
Experimental Setup
The framework was evaluated using the Nvidia PhysX engine across three environments:
- Basic Pushing: A spherical end-effector pushes a box to a target.
- Obstacle Pushing: A more complex version requiring navigation around a central barrier.
- 2D Maze: A dynamic environment used to test zero-shot adaptation to changing configurations.
Results and Discussion
L2E demonstrated significant performance gains over pure RL baselines like HER. In the basic pushing task, L2E variants reached a ~90% success rate with far fewer transitions than standard methods.
A key finding is L2E's robustness to "crude" plans. In the obstacle avoidance task, the planner provided simple "Manhattan-style" paths. The L2E agent learned to smooth these into efficient, continuous motions, discovering contact-rich strategies that the planner could not model. Furthermore, in the 2D maze task, the plan-conditioned policy allowed for zero-shot adaptation to new environment layouts, as the agent could immediately follow a new plan generated by a simple RRT planner.
Conclusion
L2E provides a robust framework for combining the strengths of planning and learning. By treating plans as informative suggestions rather than strict commands, robots can learn to navigate complex environments with high sample efficiency. Future work will focus on addressing globally misleading plans and transitioning these capabilities from simulation to real-world robotic systems.
Summary updated