Implicit Subgoal Planning with Variational Autoencoders for Long-Horizon Sparse Reward Robotic Tasks

Fangyuan Wang, Anqing Duan, Peng Zhou, Shengzeng Huo, Guodong Guo, Chenguang Yang, David Navarro-Alarcon

Introduction

Reinforcement learning (RL) has seen remarkable success in various domains, yet long-horizon tasks with sparse rewards remain a formidable challenge. In scenarios such as stacking multiple blocks or organizing a drawer, a robot may execute thousands of actions before receiving any feedback. Without intermediate guidance, the agent often fails to explore the state space effectively. Subgoal planning offers a "divide-and-conquer" solution, but existing methods frequently struggle with generating subgoals that are both physically reachable (feasible) and strategically sound (optimal).

The EISP Framework

To address these challenges, we present Explicit-Implicit Subgoal Planning (EISP). EISP is an incremental planning algorithm that combines generative modeling with value-based selection to guide robots through complex, multi-step tasks. The framework is built upon three primary components:

  1. Hybrid Subgoal Generator: Utilizing a Conditional Variational Autoencoder (CVAE), this component employs an Explicit Model (the encoder) to propose subgoals based on the current state and the final objective. It is paired with an Implicit Model (the decoder) that attempts to reconstruct the final goal from the proposed subgoal, ensuring the intermediate step is logically consistent with the ultimate task.
  2. Hindsight Sampler: To ground the planner in physical reality, EISP uses a hindsight approach. It samples valid transitions from an offline dataset of previous experiences, ensuring that the generated subgoals are within the robot's reach.
  3. Value Selector: Rather than relying on a single proposal, the generator produces multiple candidates. These are then ranked by the RL Value Function ($V$), which selects the subgoal that maximizes the expected long-term reward, ensuring an optimal path to the final goal.
The EISP architecture: A Hindsight Sampler provides feasible goal transitions, the Subgoal Generator proposes candidates, and the Value Selector identifies the optimal path.

Key Insights: Feasibility and Optimality

A core contribution of EISP is the explicit handling of the feasibility-optimality trade-off. In long-horizon planning, a subgoal that is too ambitious may be unreachable, while one that is too conservative may lead to inefficient behavior or failure to progress.

Visualization of subgoal candidates: EISP distinguishes between unfeasible, feasible but non-optimal, and optimal subgoals to ensure efficient task completion.

Our ablation studies highlight that feasibility is the primary bottleneck. Removing the Hindsight Sampler leads to a significant performance collapse, as the robot frequently attempts impossible maneuvers. Furthermore, the "implicit" reasoning step—checking if a subgoal can lead back to the final goal—provides a robust self-correction mechanism that traditional explicit planners lack.

Experimental Evaluation

EISP was evaluated across four increasingly difficult robotic manipulation tasks: Stack (3-block stacking), Push4 (sequential object pushing), OpenDrawer (obstacle removal and drawer opening), and Store (a complex sequence of opening, placing, and closing).

Compared to state-of-the-art baselines like RIS, HP, and LEAP, EISP demonstrated substantial improvements:

  • Success Rates: In the Push4 task, EISP achieved a success rate of over 70%, nearly doubling the performance of the strongest baseline (LEAP at 38%).
  • Efficiency: EISP consistently yielded higher expected returns, indicating more direct and successful task completion.
  • Real-World Robustness: The system was validated on Tiago++ and UR3 robotic platforms. EISP showed remarkable robustness, successfully re-planning subgoals in real-time when human intervention altered the environment state.

Conclusion and Future Work

EISP provides a powerful framework for long-horizon robotic planning by ensuring subgoals are both reachable and optimal. While highly effective, the system currently faces a "Goldilocks" challenge regarding subgoal distance: subgoals that are too close may cause the policy to stall, while those too far may be unsolvable for the low-level controller. Future research will explore integrating more advanced exploration techniques and applying EISP to a broader range of combinatorial decision-making problems.

Summary updated

Abstract

The challenges inherent in long-horizon tasks in robotics persist due to the typical inefficient exploration and sparse rewards in traditional reinforcement learning approaches. To address these challenges, we have developed a novel algorithm, termed Explicit-Implicit Subgoal Planning (EISP), designed to tackle long-horizon tasks through a divide-and-conquer approach. We utilize two primary criteria, feasibility and optimality, to ensure the quality of the generated subgoals. EISP consists of three components: a hybrid subgoal generator, a hindsight sampler, and a value selector. The hybrid subgoal generator uses an explicit model to infer subgoals and an implicit model to predict the final goal, inspired by way of human thinking that infers subgoals by using the current state and final goal as well as reason about the final goal conditioned on the current state and given subgoals. Additionally, the hindsight sampler selects valid subgoals from an offline dataset to enhance the feasibility of the generated subgoals. While the value selector utilizes the value function in reinforcement learning to filter the optimal subgoals from subgoal candidates. To validate our method, we conduct four long-horizon tasks in both simulation and the real world. The obtained quantitative and qualitative data indicate that our approach achieves promising performance compared to other baseline methods. These experimental results can be seen on the website \url{https://sites.google.com/view/vaesi}.
BibTeX
@article{wang2025implicit,
  author = {Fangyuan Wang and Anqing Duan and Peng Zhou and Shengzeng Huo and Guodong Guo and Chenguang Yang and David Navarro-Alarcon},
  title = {{Implicit Subgoal Planning with Variational Autoencoders for Long-Horizon Sparse Reward Robotic Tasks}},
  journal = {IEEE Trans Autom. Sci. Eng.},
  year = {2025},
  doi = {10.1109/tase.2025.3574162},
}
Figure