Hierarchical Policies for Cluttered-Scene Grasping With Latent Plans

Lirui Wang, Xiangyun Meng, Yu Xiang, Dieter Fox

TL;DR

This paper introduces Hierarchical policies for Cluttered-scene Grasping (HCG), a framework that bridges the gap between long-term motion planning and reactive control. By encoding expert grasping trajectories into a latent embedding space, the system can sample diverse "latent plans" and use a learned critic to select the safest, most efficient path to a target object amidst obstacles.

What Problem Does This Paper Address?

Robotic grasping in cluttered environments (like a packed shelf) is notoriously difficult. Traditional approaches usually fall into two camps:

  1. Open-loop Planning: These systems estimate object poses and plan a path once. They are brittle; a small error in pose estimation or a slight bump during execution leads to failure.
  2. End-to-end Reactive Policies: These map sensor data directly to motor commands. While robust to small perturbations, they often struggle with "multi-modality" (there are many ways to reach an object) and long-term constraints like avoiding complex obstacles.

The authors aim to combine the strategic foresight of planning with the robustness of closed-loop control.

Methods

The HCG framework uses a two-level hierarchy:

  • High-Level Policy (The Planner): Instead of outputting raw coordinates, this level samples latent plans.
  • Latent Embedding: During training, expert trajectories (from an OMG planner) are compressed into a 64-dimensional latent space using a Variational Autoencoder (cVAE).
  • Plan Sampler & Critic: At test time, the robot uses the cVAE to "imagine" several possible trajectories. A Plan Critic, trained via Reinforcement Learning (Q-learning), scores these plans based on their likelihood of success and collision avoidance.
  • Low-Level Policy (The Executor):
    • This policy takes the selected latent plan and the current point cloud observation to output immediate end-effector poses at a high frequency (25 Hz).
  • Policy Switching: As the robot nears the object, an "option classifier" autonomously switches control to a specialized grasping primitive (GA-DDPG) designed for high-precision terminal grasping.

The entire system operates on accumulated segmented point clouds, allowing the robot to "remember" parts of the scene it has already scanned, even if they are currently occluded.

Datasets

  • ShapeNet: Used for large-scale offline training. The authors used ~1,500 objects from 169 classes to generate 120,000 demonstration data points.
  • YCB Object and Model Set: Used for online reinforcement learning and final evaluation. This includes common household items like cracker boxes and mustard bottles.
  • Acronym: A large-scale dataset of successful grasps used to provide the "ground truth" for the expert planner during training.

Key Insights and Ideas

  • Latent Plans as Action Space: By treating a whole trajectory embedding as a "high-level action," the robot can reason about complex movements (like reaching around a box) without getting bogged down in joint-level math at the planning stage.
  • Capturing Multi-modality: The cVAE is crucial because there isn't just one "right" way to grasp an object. The latent space naturally clusters different strategies (e.g., approaching from the left vs. the right), which the critic can then evaluate based on the specific clutter in the scene.
  • Closed-Loop Planning: Unlike traditional planners that plan once, HCG can resample and re-select plans mid-motion if the environment changes or the execution drifts.
  • The "Grasping Primitive" Hand-off: The authors realized that the policy needed to reach the object is different from the policy needed to actually pinch it. Using a dedicated switching classifier to hand off control to a specialized grasper significantly boosted success rates.

Key Results

  • Simulation Success: In cluttered environments with up to 7 obstacles, HCG achieved an 81.2% success rate, outperforming state-of-the-art baselines like 6DGraspNet (67%) and Contact-GraspNet (72%).
  • Efficiency: HCG is significantly faster than traditional motion planners. Inference for plan sampling and selection takes only 0.04 seconds, compared to 1.42 seconds for some baselines.
  • Real-World Validation: The system was tested on a Franka Emika Panda arm in real-world cluttered scenes. It successfully completed 21 out of 34 attempts on 8 challenging test scenes, demonstrating that the "sim-to-real" gap is bridgeable with this hierarchical approach.

Limitations and Future Work

  • Physics Gaps: Most real-world failures were due to "slippery contacts" or small collisions that the simulator didn't model perfectly.
  • Segmentation Reliance: The system currently requires a separate, high-quality instance segmentation mask to identify the target and obstacles. The authors suggest that integrating the segmentation directly into the policy could be a future step.
  • Limited Modalities: The current model relies entirely on vision (point clouds). Future work could incorporate tactile sensing to help the robot "feel" its way through a tight clutter when vision is occluded.

Summary updated

Abstract

6D grasping in cluttered scenes is a longstanding problem in robotic manipulation. Open-loop manipulation pipelines may fail due to inaccurate state estimation, while most end-to-end grasping methods have not yet scaled to complex scenes with obstacles. In this work, we propose a new method for end-to-end learning of 6D grasping in cluttered scenes. Our hierarchical framework learns collision-free target-driven grasping based on partial point cloud observations. We learn an embedding space to encode expert grasping plans during training and a variational autoencoder to sample diverse grasping trajectories at test time. Furthermore, we train a critic network for plan selection and an option classifier for switching to an instance grasping policy through hierarchical reinforcement learning. We evaluate and analyze our method and compare against several baselines in simulation, and demonstrate that our latent planning can generalize to real-world cluttered-scene grasping tasks. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
BibTeX
@article{wang2022hierarchical,
  author = {Lirui Wang and Xiangyun Meng and Yu Xiang and Dieter Fox},
  title = {{Hierarchical Policies for Cluttered-Scene Grasping With Latent Plans}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2022},
  url = {https://openreview.net/forum?id=OiChwrGkAF},
}
Figure