Experience with Highly Automated Unmanned Aircraft Performing Complex Missions

Nimrod Rooz, Eric T. Johnson, Allen Wu, Claus Christmann, Jin-Cheol Halswijk, Girish Chowdhary, D. Michael Sobers, S. Thabasu Kannan, Wayne Pickell, Henrik B. Christophersen, Allen Tannenbaum, Jehoon Lee, Joeong Hur, Scott Kimbrell, Hal Gates, Brendan Andrus, Alison ProctorAIAA 2009 · DOI

TL;DR

This paper details the development and flight testing of an autonomous Unmanned Aerial Vehicle (UAV) system designed to perform complex surveillance missions, including searching for a target building, identifying an entry point, and entering the building to transmit imagery. The system successfully demonstrated autonomous operation for a challenging mission scenario, highlighting the integration of advanced GNC, image processing, and mission management algorithms.

What Problem Does This Paper Address?

The increasing complexity and scope of Unmanned Aerial Vehicle (UAV) operations necessitate greater autonomy, reducing the reliance on human operators for real-time decision-making. This paper addresses the challenge of developing a UAV system capable of executing a multi-stage, complex mission without human intervention. Specifically, the mission involves autonomously searching a predefined area for a target building, identifying a suitable entry point (a window), and then entering the building with a sub-vehicle to transmit imagery. This requires a sophisticated integration of perception, navigation, control, and decision-making capabilities.

Methods

The system is built around a modified Yamaha RMax helicopter, dubbed the GTMax, equipped with dual onboard computers for flight control and vision processing. Key methodological contributions include:

  • Guidance, Navigation, and Control (GNC): A 17-state extended Kalman filter for navigation and a model reference adaptive control (MRAC) system with dynamic inversion and a neural network for trajectory following.
  • Image Processing for Building and Entry Point Identification:
    • Symbol Detection: A two-stage approach using shape matching (with Halcon library) followed by feature ranking based on color and pixel ratio to identify the target building.
  • Window Detection: A hybrid algorithm combining region-based curve evolution (inspired by Chan-Vese methodology) with shape analysis to identify potential window openings. This method leverages fast level set techniques and analyzes pixel orientation differences.
  • Tracking: A system to track the identified symbol and window locations, updating their local geographical coordinates (LGC) and confidence levels based on image processor scores and navigation data.
  • Mission Management: A sophisticated mission manager orchestrates the various subsystems, autonomously making operational decisions and synchronizing tasks like searching, identifying, approaching, and deploying a ground robot.
  • Slung Load Entry: A novel approach for building entry using a 12ft boom with a rappelling mechanism to lower a ground robot into the building through a detected window. The system incorporates a bump switch for safety and a retry mechanism.
  • Ground Robot: A small rover equipped with a camera, wireless access point, and sensors for autonomous exploration of the building's interior using wall-following and wall-bouncing strategies.

Datasets

The paper does not explicitly mention the use of pre-existing datasets for training or evaluation. Instead, the system was developed and tested using a comprehensive simulation environment that included models for the GTMax, the slung load, and the ground robot. Flight tests were conducted in a real-world scenario, likely at the McKenna Mout site, Ft. Benning, as part of the International Aerial Robotics Competition (IARC).

Key Insights and Ideas

  • Integrated Autonomy is Crucial: The success of such complex missions hinges on the seamless integration of multiple autonomous systems (GNC, vision, mission management) working in concert.
  • Simulation as a Development Bedrock: The extensive use of a full closed-loop simulation environment was critical for developing and validating algorithms, especially for complex maneuvers like window tracking and slung-load entry, before flight testing.
  • Hybrid Image Processing for Robustness: Combining shape matching with feature ranking for symbol detection, and curve evolution with shape analysis for window detection, provides robustness against variations in lighting and image quality.
  • Adaptive Control for Dynamic Environments: The use of MRAC with a neural network for trajectory control allows the system to adapt to the dynamic and potentially unpredictable flight conditions encountered during complex maneuvers.
  • Careful Human-Machine Interface Design: When human operators are involved (even for monitoring), clear communication of system intent and health is paramount for effective oversight.

Key Results

  • The autonomous UAV system successfully completed the complex surveillance mission in simulation and during flight tests.
  • The window tracking algorithm demonstrated convergence of lateral estimation error to under 1.5 ft, which is sufficient for inserting a 3 ft wide ground robot.
  • The system was able to autonomously search for a target building, identify an entry point (window), and deploy a ground robot into the building.
  • The paper highlights the challenges encountered during flight tests, such as issues with brightness variance affecting symbol detection, magnetometer calibration for boom heading, and the window tracker locking onto non-window objects. Despite these, the core functionalities were demonstrated.

Limitations and Future Work

  • Image Processing Robustness: While the system demonstrated success, limitations were noted regarding brightness variance affecting symbol detection and the window tracker sometimes locking onto non-window objects or overestimating object size. Future work could focus on improving the robustness of these algorithms to a wider range of environmental conditions.
  • Boom Heading Estimation: The accuracy of the boom heading estimate, particularly due to magnetometer calibration issues, was a significant factor in some failed attempts. Further investigation and calibration of the magnetometer are suggested.
  • Entry Point Selection: The system's ability to select the "ideal" window was not always perfect, leading to failed approaches. Future work could refine the criteria for selecting the most suitable entry point.
  • Communication Reliability: The paper mentions a faulty wireless modem that failed to transmit boom heading information, highlighting the importance of robust communication links.
  • Further Flight Testing: While the system was tested, the paper implies that continued flight testing and refinement would be beneficial, especially for addressing the specific issues encountered during the IARC competition.

Summary updated

Abstract

As the roles of UAVs in the aerospace arena increase, there is much interest in increasing their ability to operate autonomously without the need of operator intervention. The following paper describes the development and testing of an autonomous UAV system capable of completing complex surveillance missions. The system has to search for its target building in a predefined search area, find an entry point on the target building and finally enter the building to transmit images from within the building back to a ground control station, all without the interaction of a human operator. Such complex missions require development of algorithms from multiple disciplines such as image processing, target tracking and estimation, and GNC(Guidance, Navigation and Control) algorithms. Furthermore a sophisticated mission manager is required to synchronize the operation of the separate subsystems and that is capable of autonomously making operational decisions. This paper will present the development of these various algorithms and flight test results of the autonomous system.
BibTeX
@inproceedings{rooz2009experience,
  author = {Nimrod Rooz and Eric T. Johnson and Allen Wu and Claus Christmann and Jin-Cheol Halswijk and Girish Chowdhary and D. Michael Sobers and S. Thabasu Kannan and Wayne Pickell and Henrik B. Christophersen and Allen Tannenbaum and Jehoon Lee and Joeong Hur and Scott Kimbrell and Hal Gates and Brendan Andrus and Alison Proctor},
  title = {{Experience with Highly Automated Unmanned Aircraft Performing Complex Missions}},
  booktitle = {AIAA},
  year = {2009},
  doi = {10.2514/6.2009-6157},
}
Figure