Approaches and Challenges in Robotic Perception for Table-top Rearrangement and Planning

Aditya Agarwal, Bipasha Sen, Shankara Narayanan V, Vishal Reddy Mandadi, Brojeshwar Bhowmick, K. Madhava KrishnaarXiv 2022 · DOI · arXiv

Abstract

Table-top Rearrangement and Planning (TTRP) is a fundamental challenge in domestic robotics, requiring systems to autonomously organize cluttered environments. This paper presents a critical analysis of the perception stack utilized during the ICRA 2022 OCRTOC challenge. We identify significant bottlenecks in scene registration, 6D pose estimation, and grasp selection, particularly when dealing with unknown objects and sensor noise. Our findings suggest that while current model-free approaches are promising, substantial improvements in computational efficiency and structural understanding are required for real-world deployment.

Introduction

The goal of TTRP is to enable a robot to perceive a disordered scene and rearrange objects into a target configuration. While motion planning is relatively mature, the perception of unknown objects in clutter remains a major hurdle. Robots in novel environments often encounter objects not present in their training sets, necessitating robust generalization. This work evaluates the standard perception pipeline against the constraints of the ICRA 2022 Open Cloud Robot Table Organization Challenge (OCRTOC), where the authors secured a third-place finish.

Methodology

The standard TTRP perception pipeline is generally decomposed into three distinct stages:

  1. Scene Registration: This involves capturing 3D point clouds from multiple viewpoints and stitching them into a coherent global frame, typically using Iterative Closest Point (ICP) algorithms.
  2. 6D Object Pose Estimation: The system determines the precise position and orientation of objects. Indirect methods are often employed, matching 2D RGB projections to 3D models via deep learning descriptors (e.g., ResNet-based features).
  3. Manipulation Point Detection: For unknown objects, "model-free" grasp generators like GraspNet-1Billion or Contact-GraspNet are used to propose grasp candidates based solely on point cloud geometry.
Overview of the TTRP perception pipeline, illustrating the transition from initial scene registration to grasp estimation and final state execution.

Critical Bottlenecks and Insights

Our analysis reveals several key challenges that hinder performance in competitive and real-world settings:

  • Latency: Current perception stacks are computationally expensive. Scene registration can take approximately 1.5 minutes, while full-scene pose estimation may exceed 5 minutes. This latency is a primary barrier to fluid human-robot interaction.
  • The "Hole" Problem: Sensor noise and occlusions often result in incomplete point clouds. Grasping algorithms frequently fail by proposing grasp points within these "holes," leading to inevitable collisions or missed grasps.
  • Structure-Agnostic Grasping: Model-free methods often lack semantic understanding. A robot might attempt to grasp a tool by its functional pivot rather than its handle because it treats all geometry as equivalent.
Visualization of dense grasp pose candidates generated during the manipulation point detection phase, highlighting the complexity of selecting stable grasps in clutter.

Conclusion and Future Directions

To overcome these limitations, we propose several research directions. First, reducing the dependency on 3D CAD models is essential for "in the wild" operation. Second, shifting from slow 3D registration to direct 3D grasp prediction from 2D RGB images could significantly reduce latency. Finally, integrating tactile feedback and reinforcement learning for grasp ranking could provide the necessary robustness to handle sensor noise and functional structural requirements that purely geometric methods currently miss.

Summary updated

Abstract

Table-top Rearrangement and Planning is a challenging problem that relies heavily on an excellent perception stack. The perception stack involves observing and registering the 3D scene on the table, detecting what objects are on the table, and how to manipulate them. Consequently, it greatly influences the system's task-planning and motion-planning stacks that follow. We present a comprehensive overview and discuss the different challenges associated with the perception module. This work is a result of our extensive involvement in the ICRA 2022 Open Cloud Robot Table Organization Challenge, in which we stood third in the final rankings.
BibTeX
@misc{agarwal2022approaches,
  author = {Aditya Agarwal and Bipasha Sen and Shankara Narayanan V and Vishal Reddy Mandadi and Brojeshwar Bhowmick and K. Madhava Krishna},
  title = {{Approaches and Challenges in Robotic Perception for Table-top Rearrangement and Planning}},
  year = {2022},
  eprint = {2205.04090},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.04090},
}
Figure