Abstract
Robotic manipulation in cluttered environments requires long-horizon planning and the ability to coordinate multiple interactions. We present the Cascaded Variational Inference Planner (CAVIN), a hierarchical model-based reinforcement learning framework that solves complex, multi-step tasks. By decoupling high-level "effects" (subgoals) from low-level "motions" (actions) through nested latent spaces, CAVIN enables robots to plan strategically over long horizons, significantly outperforming traditional "flat" generative models.
Introduction
While robots excel at isolated, single-step tasks, sequential manipulation remains a significant challenge. Tasks such as moving multiple obstacles to reach a target object suffer from the "curse of dimensionality." In continuous environments with raw visual inputs and high-dimensional action spaces, standard Model Predictive Control (MPC) often fails because random sampling rarely discovers the specific sequence of actions required for a multi-step solution. Existing generative models lack the hierarchical reasoning necessary to link a high-level strategic goal (e.g., "clear the path") to the precise low-level motor commands required to execute it.
The CAVIN Framework
CAVIN addresses these challenges through a two-level hierarchical architecture learned via Cascaded Variational Inference. This structure separates the intent of an action from its execution:
- High-Level (Meta-Dynamics): This module operates at a coarse temporal resolution. It utilizes an effect code ($c$) to predict a "subgoal" state ($s'$) several steps into the future, focusing on what should happen in the environment.
- Low-Level (Action Generator): This module takes the desired effect code ($c$) and a motion code ($z$) to produce a sequence of fine-grained primitive actions ($a$). It focuses on how to achieve the predicted effect.
- Dynamics Model: A standard model ($f$) predicts the immediate next state ($s''$) given an action, serving as a verification step for the feasibility of generated plans.
During planning, CAVIN samples in these latent spaces rather than raw action spaces. It first identifies a sequence of high-level effect codes that maximize reward (the "Effect-Level Plan") and then samples motion codes to realize those subgoals (the "Motion-Level Plan"). This temporal abstraction allows the robot to "see" further into the future without being overwhelmed by immediate motor noise.
Experimental Setup and Datasets
CAVIN was evaluated in both simulation and real-world environments using a 7-DoF Sawyer robot arm. The system processes segmented point clouds of various YCB objects via PointNet.
- Simulation Benchmarks: Three tasks of increasing complexity were tested: Clearing (removing objects from a region), Insertion (placing a target in a slot), and Crossing (navigating a cluttered "bridge").
- Real-World Transfer: The system was deployed on a physical Sawyer robot using a Kinect2 sensor to demonstrate "sim-to-real" transfer without explicit fine-tuning.
Key Results
CAVIN demonstrates significant improvements over state-of-the-art "flat" baselines like SeCTAR and CVAE-MPC:
- Performance: In the complex "Crossing" task, CAVIN achieved a success rate approximately 11-14% higher than the next best method.
- Robustness to Sparse Rewards: While most RL models fail without constant feedback, CAVIN’s performance dropped by less than 5% when switched to sparse rewards, where feedback is only provided upon task completion.
- Scalability: CAVIN maintained high success rates as the number of objects in the scene increased from 1 to 5, whereas baseline performance plummeted.
- Real-World Success: On the physical robot, CAVIN achieved success rates of 93.3% for Clearing, 73.3% for Insertion, and 80.0% for Crossing.
Conclusion and Future Work
By learning latent spaces through variational inference, CAVIN ensures that the robot only samples physically plausible behaviors, making the search space for complex planning manageable. However, the model currently relies on segmented point clouds, which may be a bottleneck in highly occluded scenes. Future work will explore expanding the action generator beyond planar pushing to 6-DoF manipulation (picking and placing) and investigating curiosity-driven data collection to improve training efficiency.
Summary updated