Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation

Kuan Fang, Yuke Zhu, Animesh Garg, Silvio Savarese, Li Fei-Fei

Abstract

Robotic manipulation in cluttered environments requires long-horizon planning and the ability to coordinate multiple interactions. We present the Cascaded Variational Inference Planner (CAVIN), a hierarchical model-based reinforcement learning framework that solves complex, multi-step tasks. By decoupling high-level "effects" (subgoals) from low-level "motions" (actions) through nested latent spaces, CAVIN enables robots to plan strategically over long horizons, significantly outperforming traditional "flat" generative models.

Introduction

While robots excel at isolated, single-step tasks, sequential manipulation remains a significant challenge. Tasks such as moving multiple obstacles to reach a target object suffer from the "curse of dimensionality." In continuous environments with raw visual inputs and high-dimensional action spaces, standard Model Predictive Control (MPC) often fails because random sampling rarely discovers the specific sequence of actions required for a multi-step solution. Existing generative models lack the hierarchical reasoning necessary to link a high-level strategic goal (e.g., "clear the path") to the precise low-level motor commands required to execute it.

The CAVIN Framework

CAVIN addresses these challenges through a two-level hierarchical architecture learned via Cascaded Variational Inference. This structure separates the intent of an action from its execution:

  1. High-Level (Meta-Dynamics): This module operates at a coarse temporal resolution. It utilizes an effect code ($c$) to predict a "subgoal" state ($s'$) several steps into the future, focusing on what should happen in the environment.
  2. Low-Level (Action Generator): This module takes the desired effect code ($c$) and a motion code ($z$) to produce a sequence of fine-grained primitive actions ($a$). It focuses on how to achieve the predicted effect.
  3. Dynamics Model: A standard model ($f$) predicts the immediate next state ($s''$) given an action, serving as a verification step for the feasibility of generated plans.
The CAVIN hierarchical architecture decoupling Meta-Dynamics from the Action Generator.

During planning, CAVIN samples in these latent spaces rather than raw action spaces. It first identifies a sequence of high-level effect codes that maximize reward (the "Effect-Level Plan") and then samples motion codes to realize those subgoals (the "Motion-Level Plan"). This temporal abstraction allows the robot to "see" further into the future without being overwhelmed by immediate motor noise.

Experimental Setup and Datasets

CAVIN was evaluated in both simulation and real-world environments using a 7-DoF Sawyer robot arm. The system processes segmented point clouds of various YCB objects via PointNet.

  • Simulation Benchmarks: Three tasks of increasing complexity were tested: Clearing (removing objects from a region), Insertion (placing a target in a slot), and Crossing (navigating a cluttered "bridge").
  • Real-World Transfer: The system was deployed on a physical Sawyer robot using a Kinect2 sensor to demonstrate "sim-to-real" transfer without explicit fine-tuning.
Real-world experimental setup with a Sawyer robot and Kinect2 sensor, alongside planning visualizations.

Key Results

CAVIN demonstrates significant improvements over state-of-the-art "flat" baselines like SeCTAR and CVAE-MPC:

  • Performance: In the complex "Crossing" task, CAVIN achieved a success rate approximately 11-14% higher than the next best method.
  • Robustness to Sparse Rewards: While most RL models fail without constant feedback, CAVIN’s performance dropped by less than 5% when switched to sparse rewards, where feedback is only provided upon task completion.
  • Scalability: CAVIN maintained high success rates as the number of objects in the scene increased from 1 to 5, whereas baseline performance plummeted.
  • Real-World Success: On the physical robot, CAVIN achieved success rates of 93.3% for Clearing, 73.3% for Insertion, and 80.0% for Crossing.

Conclusion and Future Work

By learning latent spaces through variational inference, CAVIN ensures that the robot only samples physically plausible behaviors, making the search space for complex planning manageable. However, the model currently relies on segmented point clouds, which may be a bottleneck in highly occluded scenes. Future work will explore expanding the action generator beyond planar pushing to 6-DoF manipulation (picking and placing) and investigating curiosity-driven data collection to improve training efficiency.

Summary updated

Abstract

The fundamental challenge of planning for multi-step manipulation is to find effective and plausible action sequences that lead to the task goal. We present Cascaded Variational Inference Planner (...
BibTeX
@inproceedings{fang2019dynamics,
  author = {Kuan Fang and Yuke Zhu and Animesh Garg and Silvio Savarese and Li Fei-Fei},
  title = {{Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation}},
  booktitle = {Conference on Robot Learning},
  year = {2019},
}
Figure