Mission-level Robustness with Rapidly-deployed, Autonomous Aerial Vehicles by Carnegie Mellon Team Tartan at MBZIRC 2020

Anish Bhattacharya, Akshit Gandhi, Lukas Merkle, Rohan Tiwari, Karun Warrior, Stanley Winata, Andrew Saba, Kevin Zhang, Oliver Kroemer, Sebastian Scherer

Abstract

For robotic systems to succeed in high risk, real-world situations, they have to be quickly deployable and robust to environmental changes, under-performing hardware, and mission subtask failures. These robots are often designed to consider a single sequence of mission events, with complex algorithms lowering individual subtask failure rates under some critical constraints. Our approach utilizes common techniques in vision and control, and encodes robustness into mission structure through outcome monitoring and recovery strategies. In addition, our system infrastructure enables rapid deployment and requires no central communication. This report also includes lessons in rapid field robotic development and testing. We developed and evaluated our systems through real-robot experiments at an outdoor test site in Pittsburgh, Pennsylvania, USA, as well as in the 2020 Mohamed Bin Zayed International Robotics Challenge. All competition trials were completed in fully autonomous mode without RTK-GPS. Our system placed fourth in Challenge 2 and seventh in the Grand Challenge, with notable achievements such as popping five balloons (Challenge 1), successfully picking and placing a block (Challenge 2), and dispensing the most water onto an outdoor, real fire with an autonomous UAV (Challenge 3).

Introduction

In controlled laboratory settings, autonomous systems often achieve near-perfect task success rates. However, transitioning to high-stakes, outdoor environments like the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) reveals the fragility of these systems. In the field, a 1% failure rate in a single subtask—such as target detection or landing—compounds across the mission, often leading to total failure.

Team Tartan from Carnegie Mellon University argues that Mission-level Robustness is more critical for field robotics than the perfection of any individual algorithm. Rather than focusing solely on lowering the failure rates of specific subtasks through complex, often fragile methods, their approach designs the mission structure to expect and recover from failures. By integrating outcome monitoring and explicit recovery strategies, the system maintains operational continuity despite faulty hardware, unpredictable weather, or sensor noise.

System Architecture and Methodology

The team utilized a standardized hardware platform based on the DJI M210 V2 for three distinct challenges: aerial object pursuit, wall construction, and firefighting. The system architecture was designed for rapid deployment and modularity.

Perception and Control

The vision pipeline combined deep learning with classical computer vision. TinyYOLOv3, optimized via Intel OpenVINO, provided general object detection. For high-precision interactions, the team employed classical HSV clustering and k-means. A key component of their local interaction strategy was Image-Based Visual Servoing (IBVS). IBVS maps pixel-level errors directly to velocity commands, significantly reducing the system's sensitivity to camera calibration errors and mechanical offsets.

The software architecture integrates custom sensors with the DJI SDK, feeding into a central state machine that manages trajectory and pose control.

Adaptive Learning

To handle varying environmental conditions, the team used custom-trained datasets for balloons and foam balls, ensuring reliability in diverse lighting. For the construction challenge, they implemented an on-site adaptive learning approach using DBSCAN (a density-based clustering algorithm). This allowed the drone to automatically learn the color and mask of the placement structure during an initial flight, adapting to the specific arena lighting without manual intervention.

Robust Mission Logic: The Detect-Attempt-Recover Cycle

The core of the system's resilience is its state machine, which encodes "Outcome Monitoring." Every action is followed by a verification step. If a subtask fails—for instance, if a drone attempts to pick up a block but the sensor does not detect a successful attachment—the state machine triggers a recovery loop rather than halting the mission.

The mission state machine incorporates explicit recovery paths, such as returning to a global search position if a target is lost or a detection timeout occurs.

This "Detect-Attempt-Recover" cycle was vital in the balloon-popping task. High-rate maneuvers often caused the drone to lose sight of the target. The recovery logic enabled the UAV to autonomously fly back to its last known "good" position to re-acquire the target, turning a potential mission failure into a successful completion.

Experimental Results

The effectiveness of this robust architecture was demonstrated at MBZIRC 2020:

  • Challenge 1 (Aerial Objects): The team successfully popped five balloons. While the initial "pop sequence" had a 60% success rate, the recovery logic ensured a 100% eventual completion rate.
  • Challenge 2 (Construction): Placed 4th overall. The team was one of only three to achieve fully autonomous pick-and-place operations, with the IBVS system showing 100% convergence during alignment.
  • Challenge 3 (Firefighting): The system dispensed the most water of any team onto a real outdoor fire, completing eight refill-and-extinguish cycles within a 15-minute trial.
  • Grand Challenge: Despite hardware issues, the team placed 7th, validating the software's ability to handle real-world degradation.

Discussion and Future Work

A major takeaway from the competition was the value of simplicity. The team found that a basic USB fisheye camera often outperformed high-end gimbal cameras, which suffered from software latency. Furthermore, their automated "watchdog" pipeline allowed for rapid deployment, enabling seven mission runs in under 20 minutes—a pace that manual systems could not match.

Future work will address the limitations of monocular vision for 3D localization and the "open-loop" drift encountered during the final centimeters of descent in manipulation tasks. Additionally, the team aims to scale these robust state machines to coordinate multi-agent systems involving multiple UAVs and ground vehicles.

Summary updated

Abstract

For robotic systems to succeed in high risk, real-world situations, they have to be quickly deployable and robust to environmental changes, under-performing hardware, and mission subtask failures. These robots are often designed to consider a single sequence of mission events, with complex algorithms lowering individual subtask failure rates under some critical constraints. Our approach utilizes common techniques in vision and control, and encodes robustness into mission structure through outcome monitoring and recovery strategies. In addition, our system infrastructure enables rapid deployment and requires no central communication. This report also includes lessons in rapid field robotic development and testing. We developed and evaluated our systems through real-robot experiments at an outdoor test site in Pittsburgh, Pennsylvania, USA, as well as in the 2020 Mohamed Bin Zayed International Robotics Challenge. All competition trials were completed in fully autonomous mode without RTK-GPS. Our system placed fourth in Challenge 2 and seventh in the Grand Challenge, with notable achievements such as popping five balloons (Challenge 1), successfully picking and placing a block (Challenge 2), and dispensing the most water onto an outdoor, real fire with an autonomous UAV (Challenge 3).
BibTeX
@inproceedings{bhattacharya2022mission,
  author = {Anish Bhattacharya and Akshit Gandhi and Lukas Merkle and Rohan Tiwari and Karun Warrior and Stanley Winata and Andrew Saba and Kevin Zhang and Oliver Kroemer and Sebastian Scherer},
  title = {{Mission-level Robustness with Rapidly-deployed, Autonomous Aerial Vehicles by Carnegie Mellon Team Tartan at MBZIRC 2020}},
  booktitle = {Field},
  year = {2022},
  doi = {10.55417/fr.2022007},
}
Figure