Introduction
Reinforcement learning (RL) has seen remarkable success in various domains, yet long-horizon tasks with sparse rewards remain a formidable challenge. In scenarios such as stacking multiple blocks or organizing a drawer, a robot may execute thousands of actions before receiving any feedback. Without intermediate guidance, the agent often fails to explore the state space effectively. Subgoal planning offers a "divide-and-conquer" solution, but existing methods frequently struggle with generating subgoals that are both physically reachable (feasible) and strategically sound (optimal).
The EISP Framework
To address these challenges, we present Explicit-Implicit Subgoal Planning (EISP). EISP is an incremental planning algorithm that combines generative modeling with value-based selection to guide robots through complex, multi-step tasks. The framework is built upon three primary components:
- Hybrid Subgoal Generator: Utilizing a Conditional Variational Autoencoder (CVAE), this component employs an Explicit Model (the encoder) to propose subgoals based on the current state and the final objective. It is paired with an Implicit Model (the decoder) that attempts to reconstruct the final goal from the proposed subgoal, ensuring the intermediate step is logically consistent with the ultimate task.
- Hindsight Sampler: To ground the planner in physical reality, EISP uses a hindsight approach. It samples valid transitions from an offline dataset of previous experiences, ensuring that the generated subgoals are within the robot's reach.
- Value Selector: Rather than relying on a single proposal, the generator produces multiple candidates. These are then ranked by the RL Value Function ($V$), which selects the subgoal that maximizes the expected long-term reward, ensuring an optimal path to the final goal.
Key Insights: Feasibility and Optimality
A core contribution of EISP is the explicit handling of the feasibility-optimality trade-off. In long-horizon planning, a subgoal that is too ambitious may be unreachable, while one that is too conservative may lead to inefficient behavior or failure to progress.
Our ablation studies highlight that feasibility is the primary bottleneck. Removing the Hindsight Sampler leads to a significant performance collapse, as the robot frequently attempts impossible maneuvers. Furthermore, the "implicit" reasoning step—checking if a subgoal can lead back to the final goal—provides a robust self-correction mechanism that traditional explicit planners lack.
Experimental Evaluation
EISP was evaluated across four increasingly difficult robotic manipulation tasks: Stack (3-block stacking), Push4 (sequential object pushing), OpenDrawer (obstacle removal and drawer opening), and Store (a complex sequence of opening, placing, and closing).
Compared to state-of-the-art baselines like RIS, HP, and LEAP, EISP demonstrated substantial improvements:
- Success Rates: In the Push4 task, EISP achieved a success rate of over 70%, nearly doubling the performance of the strongest baseline (LEAP at 38%).
- Efficiency: EISP consistently yielded higher expected returns, indicating more direct and successful task completion.
- Real-World Robustness: The system was validated on Tiago++ and UR3 robotic platforms. EISP showed remarkable robustness, successfully re-planning subgoals in real-time when human intervention altered the environment state.
Conclusion and Future Work
EISP provides a powerful framework for long-horizon robotic planning by ensuring subgoals are both reachable and optimal. While highly effective, the system currently faces a "Goldilocks" challenge regarding subgoal distance: subgoals that are too close may cause the policy to stall, while those too far may be unsolvable for the low-level controller. Future research will explore integrating more advanced exploration techniques and applying EISP to a broader range of combinatorial decision-making problems.
Summary updated