TL;DR
Researchers from the University of Bonn developed "Jelly," an autonomous micro aerial vehicle (MAV) that uses TPU-accelerated computer vision and specialized flexible "tentacles" to pop balloons. During the MBZIRC 2020 competition, the system successfully popped all five target balloons in just 81 seconds of flight time, demonstrating high-speed autonomous perception and interaction.
What Problem Does This Paper Address?
Autonomous aerial manipulation—interacting with objects while flying—is a significant challenge in robotics. The Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2020 task required drones to locate and pop five green balloons (60 cm diameter) tethered to 2.5m poles in a large outdoor arena (90m x 40m).
To succeed, a robot needs more than just flight stability. It requires:
- Robust Perception: Detecting small targets against complex outdoor backgrounds and varying light.
- Precise Localization: Estimating the 3D position of targets accurately enough to strike them.
- Physical Resilience: A mechanism to pop the balloon without the impact or entanglement causing the drone to crash.
- Onboard Computing: All processing must happen on the drone to minimize latency.
Methods
The authors presented an integrated system called "Jelly," based on the DJI Matrice 100 platform.
Hardware Design
The most striking feature is the "tentacles": four 1.4m long flexible rods tipped with spiked hemispheres. These are designed to be detachable; if a tentacle gets snagged on a pole, it releases, allowing the MAV to continue its mission rather than crashing.
Perception Pipeline
- CNN-based Detection: They used a modified ResNet-18 architecture for semantic segmentation, specifically trained to detect balloon outlines.
- Edge TPU Acceleration: To achieve real-time performance (45ms per frame), the network was quantized to 8-bit integers and run on a Google Edge TPU USB Accelerator.
- Distance Estimation: Instead of relying solely on noisy stereo depth data at long ranges, the team used the known physical size of the balloons (60cm). By measuring the radius of the detected balloon in the 2D image, they could geometrically calculate its 3D distance.
Filtering and Control
- Ray-casting Filter: To aggregate multiple detections into a stable "balloon hypothesis," they used a distance-to-ray metric. This proved more robust than standard Euclidean distance, which is often skewed by depth noise.
- Trajectory Generation: The MAV uses a Model Predictive Controller (MPC) to generate third-order, time-optimal trajectories (limiting jerk and acceleration) to ensure smooth, high-speed approaches.
Datasets
The vision system was trained using a mix of data:
- Synthetic Data: 10,000 images generated using the EasyPBR renderer, featuring 3D balloon models in various HDR environments.
- Real-world Data: 300 real images for initial training, supplemented by 250 additional annotated images captured on-site during competition rehearsals to account for local lighting and background conditions.
Key Insights and Ideas
- Flexibility is Safety: While other teams used rigid popping mechanisms, Jelly’s flexible tentacles allowed for "imperfect" positioning. The MAV didn't need to hit the balloon's dead center; it just had to drag its tentacles across the surface.
- The "Star-Shaped" Strategy: In early runs, the MAV flew directly from one balloon to the next. However, turning over a balloon caused the dangling tentacles to swing away from the target. In the final "Grand Challenge" run, they programmed the MAV to return toward the center of the arena after each pop. This ensured it always approached balloons in a straight line, significantly increasing the success rate.
- Quantization Matters: Moving from floating-point to 8-bit integer quantization allowed the vision model to run on low-power hardware (Edge TPU) without a noticeable drop in detection quality.
Key Results
- Speed: In the final Grand Challenge run, Jelly popped all five balloons in 1 minute and 21 seconds.
- Reliability: The system achieved a 100% pop rate in the final run, with every balloon punctured on the first attempt.
- Detection Range: The fine-tuned vision pipeline was able to detect balloons at distances of up to 50 meters.
- Competition Standing: The team placed 2nd in the Grand Challenge and 5th in the specific balloon-popping sub-challenge.
Limitations and Future Work
- Geofencing Issues: The team used a simple rectangular geofence. During one run, the MAV got stuck because the actual arena was non-convex (L-shaped), and the drone's safety logic prevented it from navigating a corner correctly.
- Hardware Constraints: While the Edge TPU was efficient, the authors noted that its architecture is more restrictive than platforms like NVIDIA Jetson, limiting the types of neural networks that can be deployed.
- Static Targets: The current system assumes balloons are static. Future work could involve extending the perception and trajectory logic to track and intercept moving targets.
Summary updated