Autonomous Marker-less Rapid Aerial Grasping

Erik Bauer, Barnabas Gavin Cangan, Robert K. Katzschmann

TL;DR

Researchers at ETH Zurich have developed a vision-based system that allows drones to perform rapid "swoop" grasping of novel objects without the need for artificial markers or pre-defined 3D models. By combining Mask R-CNN instance segmentation with geometry-based grasp planning, the system achieves grasping success rates comparable to high-end motion capture systems, marking a significant step toward autonomous aerial manipulation in unstructured environments like warehouses.

What Problem Does This Paper Address?

Aerial grasping—the ability of a drone to pick up an object while in flight—is a high-stakes robotics challenge. Traditionally, researchers have relied on two "crutches" to make this work:

  1. Motion Capture (MoCap): Using expensive external camera arrays to track the drone and the object with millimeter precision.
  2. Fiducial Markers: Attaching high-contrast tags (like QR codes or ArUco markers) to every object the robot might need to touch.

Neither solution scales to the real world. You can't put markers on every item in a massive warehouse, and you certainly can't install MoCap systems everywhere. Furthermore, rapid "swooping" maneuvers (where the drone doesn't stop to hover) introduce motion blur and tracking uncertainties that make standard vision systems fail. This paper aims to bridge the gap by creating a marker-less, vision-only pipeline that works in real-time.

Methods

The authors deployed their system on RAPTOR, a quadcopter equipped with a soft Fin Ray gripper that naturally conforms to different shapes. The technical pipeline consists of three main stages:

  1. Perception & Segmentation: The drone carries an Intel RealSense D455 RGB-D camera. The system uses Mask R-CNN (via the Detectron2 framework) to identify objects in the RGB frame and generate a precise pixel-level mask.
  2. Point Cloud Generation: This mask is applied to the depth data to extract a dense 3D point cloud of just the target object. Standard cleaning techniques (radius outlier removal and voxel downsampling) are used to refine the data.
  3. Geometry-Based Grasp Planning: Instead of using a pre-known model, the system assumes the object is roughly symmetric. It estimates the object's centroid, "mirrors" the visible point cloud to estimate the hidden back side, and then calculates optimal contact points for the gripper using a "cutting plane" method.
The grasp planning pipeline: from raw point cloud to identified grasping candidates.

To handle the heavy computational load of Mask R-CNN, the team implemented a hybrid architecture. An onboard Nvidia Jetson Nano compresses the video (using a clever mix of lossy JPG for color and lossless PNG for depth to preserve accuracy) and streams it to an offboard PC with an RTX 3070 for processing.

Datasets

The Mask R-CNN model was pre-trained on the COCO (Common Objects in Context) dataset, which allows it to recognize a wide variety of everyday items out of the box. For evaluation, the researchers performed real-world tests using a Teddy Bear and a PET Bottle, comparing their vision-based results against a baseline established by a Vicon motion capture system.

Key Insights and Ideas

  • Workspace Expansion: The authors found that Mask R-CNN significantly outperformed lighter architectures like YOLOv5 in terms of range. While YOLOv5 struggled beyond 0.6 meters for small objects, Mask R-CNN could precisely segment a bottle from up to 2.25 meters away. This gives the drone much more time to plan its approach.
  • Decoupling Perception from Control: The vision pipeline is explicitly decoupled from the drone's motion planning. This modularity means the vision system could be "dropped in" to other robotic platforms with minimal changes.
  • The Symmetry Shortcut: By assuming axial symmetry, the system can plan grasps on "novel" objects it has never seen before, avoiding the need for a library of 3D CAD models.
  • Communication Efficiency: To maintain real-time performance over Wi-Fi, the researchers used a custom ZeroMQ/Protobuf stack, which proved more efficient than standard ROS (Robot Operating System) overhead.
The system architecture showing the dataflow between the onboard drone components and the offboard processing PC.

Key Results

The system was validated through 36 grasp attempts for each object:

  • Success Rates: The system achieved an 89% success rate for the teddy bear and 64% for the PET bottle.
  • Comparison to Baseline: The vision-based system reached 94.5% of the performance of the MoCap baseline for the teddy bear. For the bottle, the vision system actually slightly outperformed the recorded MoCap baseline (64% vs 61%), likely due to the specific size of the test objects.
  • Latency: The total round-trip time from image capture to grasp command was approximately 220ms, with a mean network transit time of only 41ms.

Limitations and Future Work

While impressive, the system has clear areas for growth:

  • Offboard Dependency: The reliance on an offboard PC limits the drone to areas with strong Wi-Fi. Future work aims to optimize the models for full onboard execution.
  • Symmetry Assumption: The current grasp planner assumes objects are symmetric. It would likely struggle with highly irregular or complex shapes.
  • Motion Blur: During high-speed swoops, motion blur still prevents the system from providing constant closed-loop feedback in the final milliseconds of the grasp.
  • Outdoor Environments: The researchers plan to integrate visual-inertial odometry to allow the system to operate in unstructured outdoor environments where even the drone's own position isn't known via MoCap.

Summary updated

Abstract

In a future with autonomous robots, visual and spatial perception is of utmost importance for robotic systems. Particularly for aerial robotics, there are many applications where utilizing visual perception is necessary for any real-world scenarios. Robotic aerial grasping using drones promises fast pick-and-place solutions with a large increase in mobility over other robotic solutions. Utilizing Mask R-CNN scene segmentation (detectron2), we propose a vision-based system for autonomous rapid aerial grasping which does not rely on markers for object localization and does not require the appearance of the object to be previously known. Combining segmented images with spatial information from a depth camera, we generate a dense point cloud of the detected objects and perform geometry-based grasp planning to determine grasping points on the objects. In real-world experiments on a dynamically grasping aerial platform, we show that our system can replicate the performance of a motion capture system for object localization up to 94.5 % of the baseline grasping success rate. With our results, we show the first use of geometry-based grasping techniques with a flying platform and aim to increase the autonomy of existing aerial manipulation platforms, bringing them further towards real-world applications in warehouses and similar environments.
BibTeX
@inproceedings{bauer2023autonomous,
  author = {Erik Bauer and Barnabas Gavin Cangan and Robert K. Katzschmann},
  title = {{Autonomous Marker-less Rapid Aerial Grasping}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=p4r8jcbJw0},
}

Loading related content...

Figure