Autonomous Marker-Less Rapid Aerial Grasping

Erik Bauer, Barnabas Gavin Cangan, Robert K. Katzschmann

TL;DR

This paper presents a vision-based system for autonomous aerial grasping that doesn't require markers on objects. By combining Mask R-CNN for object segmentation with depth camera data for point cloud generation and geometry-based grasp planning, their system enables rapid, marker-less grasping with drones, achieving performance comparable to motion capture systems.

What Problem Does This Paper Address?

Autonomous robots, especially aerial ones, need robust visual perception to interact with their environment. While drones offer increased mobility, their practical applications are limited by their ability to locate and grasp objects. Traditional methods often rely on motion capture systems or fiducial markers, which are impractical for real-world scenarios. Monocular vision alone struggles with depth perception, and existing marker-less approaches often require prior knowledge of object shapes or dimensions. This paper tackles the challenge of enabling drones to autonomously identify, localize, and grasp diverse objects without any pre-existing markers or object models, paving the way for more versatile aerial manipulation.

Methods

The core of the system involves a multi-stage pipeline:

  • Instance Segmentation: Mask R-CNN (specifically using the Detectron2 framework) is employed to segment target objects from RGB images. This choice is motivated by its proven ability to handle small objects at greater distances compared to simpler detection methods, thus expanding the robot's operational workspace.
  • Point Cloud Generation: The segmentation mask is used to filter points from a corresponding depth image captured by an Intel RealSense D455 camera. This process creates a dense point cloud of the segmented object. Post-processing includes radius outlier removal and voxel downsampling for noise reduction and computational efficiency.
  • Geometry-Based Grasp Planning: An adapted geometry-based grasp planning strategy is used. This involves estimating the object's centroid and pose, duplicating and rotating the point cloud to account for symmetry (assuming objects are roughly axially symmetric), and then identifying candidate grasp points by intersecting the point cloud with a cutting plane. This approach is designed to be suitable for objects that are under 500g and fit within the drone's gripper.
  • System Architecture: The system is designed with modularity in mind. An onboard Nvidia Jetson Nano captures and compresses RGB-D data, which is then sent to an offboard desktop PC for intensive processing (segmentation, grasp planning). Communication between onboard and offboard components is handled via ZeroMQ for synchronization. The results are then sent back to the drone's flight controller for execution.

Datasets

The paper does not explicitly mention training datasets for Mask R-CNN. However, the target objects used for evaluation are stated to be from the COCO dataset, which was used for training the object detection model. For evaluation, they used real-world experiments with specific objects like a teddy bear and a PET bottle.

Key Insights and Ideas

  • Marker-less Grasping at a Distance: The system demonstrates that robust object segmentation at significant distances (up to 2.25m) is achievable with Mask R-CNN, which is crucial for enabling rapid aerial approaches and expanding the drone's operational range.
  • Bridging the Gap for Geometry-Based Grasping: This work successfully transfers geometry-based grasp planning techniques, previously used with static manipulators, to a dynamic aerial platform. This is significant because it leverages established, reliable grasp planning methods without requiring complex learning for grasp pose generation.
  • Decoupling Perception from Motion: The vision system is designed to be independent of the drone's motion planning. This modularity makes it easier to integrate into existing aerial platforms and adapt for different tasks beyond grasping.
  • Real-time Performance with Offboard Processing: While onboard processing is explored, the paper highlights the necessity of offboard computation for achieving real-time performance with demanding tasks like Mask R-CNN segmentation, especially given current onboard hardware limitations.

Key Results

  • Grasping Success Rates: The system achieved an 89% grasping success rate for a teddy bear and a 64% success rate for a PET bottle.
  • Comparison to Baseline: The grasping success rates were comparable to previous work using motion capture systems for localization, with no significant decrease observed. For the teddy bear, the success rate was 94% of the baseline, and for the bottle, it was 61% of the baseline.
  • Localization Accuracy: The mean squared error (MSE) for the teddy bear was 6.2 cm (x-axis), 2.9 cm (y-axis), and 1.7 cm (z-axis). For the bottle, the MSE was 1.3 cm (x-axis), 1.3 cm (y-axis), and 2.7 cm (z-axis).
  • Image Processing Latency: The camera image publisher (capture, compress, send) takes approximately 100 ms. Offboard processing (decompression, segmentation, grasp planning) takes about 120 ms. The overall system achieves real-time performance suitable for various scenarios.

Limitations and Future Work

  • Object Assumptions: The grasp planning algorithm assumes objects are roughly axially symmetric and fit within the gripper (under 500g). This limits the types of objects that can be reliably grasped.
  • Thickness and Symmetry: Thicker objects and those with less symmetry pose challenges for the current grasp planning approach, as it relies on the visible point cloud and symmetry assumptions.
  • Motion Blur: While the system achieves real-time performance, rapid swooping motions can introduce motion blur, which is not explicitly handled and can affect precision.
  • Future Work: The authors suggest extending the system to handle dynamic grasping in unstructured outdoor environments using visual-inertial odometry for self-localization. They also aim to enable grasping of larger, more strongly asymmetric objects and explore learning-based methods for generating more comprehensive 3D point clouds.

Summary updated

Abstract

In a future with autonomous robots, visual and spatial perception is of utmost importance for robotic systems. Particularly for aerial robotics, there are many applications where utilizing visual perception is necessary for any real-world scenarios. Robotic aerial grasping using drones promises fast pick-and-place solutions with a large increase in mobility over other robotic solutions. Utilizing Mask R-CNN scene segmentation (detectron2), we propose a vision-based system for autonomous rapid aerial grasping which does not rely on mark-ers for object localization and does not require the appearance of the object to be previously known. Combining segmented images with spatial information from a depth camera, we generate a dense point cloud of the detected objects and perform geometry - based grasp planning to determine grasping points on the objects. In real-world experiments on a dynamically grasping aerial platform, we show that our system can replicate the performance of a motion capture system for object local-ization up to 94.5 % of the baseline grasping success rate. With our results, we show the first use of geometry-based grasping techniques with a flying platform and aim to increase the autonomy of existing aerial manipulation platforms, bringing them further towards real-world applications in warehouses and similar environments. † † Code: https://github.com/srl-ethz/detectron-realsense
BibTeX
@inproceedings{bauer2023autonomous,
  author = {Erik Bauer and Barnabas Gavin Cangan and Robert K. Katzschmann},
  title = {{Autonomous Marker-Less Rapid Aerial Grasping}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=p4r8jcbJw0},
}
Figure