Online Safe Flight Control Method Based on Constraint Reinforcement Learning

Jiawei Zhao, Haotian Xu, Zhaolei Wang, Tao ZhangDrones 2024 · DOI

TL;DR

This paper introduces a novel online safe flight control method for UAVs that combines constrained reinforcement learning (RL) with meta-learning. The approach ensures flight safety by incorporating Lyapunov stability requirements and limiting control surface deflections, achieving a perfect score in simulations and significantly improving adaptability to environmental changes.

What Problem Does This Paper Address?

Unmanned Aerial Vehicles (UAVs) are becoming increasingly complex and are deployed in challenging environments. Traditional control methods often require precise models of the aircraft, which are difficult to obtain in practice due to nonlinearities and uncertainties. Reinforcement learning offers a data-driven approach to control, but standard RL methods lack theoretical guarantees for safety and reliability, making them unsuitable for critical applications like flight control. Specifically, ensuring that a UAV maintains stable flight and avoids hazardous maneuvers (like excessive control surface deflections) while adapting to dynamic environmental conditions (like wind or changing aerodynamic parameters) is a significant challenge.

Methods

The proposed method employs a two-stage approach:

  1. Offline Training with Constrained Reinforcement Learning (ESB-CPO):
    • The core of the offline training uses an Extra Safety Budget Constrained Policy Optimization (ESB-CPO) algorithm. This algorithm is built upon constrained policy optimization (CPO) and incorporates Lyapunov stability requirements.
  2. It introduces an "extra safety budget" to manage constraints, aiming to maximize rewards while ensuring that safety-critical costs (like excessive rudder deflection) remain below a defined threshold.
  3. Adaptive factors and Lyapunov Advantage Estimation (LAE) are used to balance exploration and exploitation while adhering to safety constraints.
  4. A Python Flight Mechanics Engine (PyFME) based simulation environment is developed for this offline training.
  5. Online Learning with Meta-Learning:
    • After offline training, a condition-triggered meta-learning online learning method is employed.
  6. This method uses real-time interactive data from online flights to fine-tune the control law.
  7. Meta-learning allows the system to rapidly adapt and learn new tasks with minimal training samples, enabling it to adjust the control policy online based on observed conditions.

Datasets

The paper does not explicitly mention using pre-existing datasets. Instead, it relies on a custom-built simulation environment using PyFME for generating training and testing data. This environment models aircraft dynamics, atmospheric conditions, and aerodynamic parameters.

Key Insights and Ideas

  • Safety as a First-Class Constraint: The core innovation is integrating explicit safety constraints (Lyapunov stability, limited control surface deflection) directly into the RL optimization process, moving beyond just maximizing performance.
  • Adaptive Safety Budgets: The ESB-CPO algorithm's ability to adapt safety budgets and constraints dynamically is crucial for balancing exploration in the early stages with strict adherence to safety later on.
  • Meta-Learning for Real-World Adaptability: The combination of meta-learning with RL allows the control policy to adapt to unforeseen environmental changes (wind, parameter variations) in real-time, a critical step towards robust autonomous flight.
  • Simulation-to-Reality Gap Mitigation: While the study is simulation-based, the meta-learning component is designed to bridge the gap by enabling rapid adaptation to new tasks and environments, a step towards real-world deployment.

Key Results

  • Perfect Online Performance: The proposed online safe flight control method achieved an overall score of 100 points in attitude angle control tasks in simulations.
  • Improved Adaptability: Online learning improved the adaptability of attitude control to comprehensive errors (aerodynamic parameters, wind) by 21% compared to offline learning.
  • Reduced Control Surface Deflection: The method effectively limits rudder deflection, contributing to smoother flight and enhanced safety, as demonstrated by comparisons with TRPO and SAC.
  • Real-time Performance: The online control algorithm exhibits a single computation time of approximately 1.5 ms, meeting real-time requirements.

Limitations and Future Work

  • Simulation-Based Study: The current research is conducted solely in a simulated environment. The authors acknowledge that this may not fully capture the complexities and uncertainties of real-world scenarios.
  • Complex Scenarios: Future work should involve experiments in more complex scenarios, such as those with communication interference or motor faults, to further test and improve control strategies.
  • Diverse Aircraft Models: The authors plan to extend their research to various aircraft models to translate the findings into practical applications.

Summary updated

Abstract

UAVs are increasingly prominent in the competition for space due to their multiple characteristics, such as strong maneuverability, long flight distance, and high survivability. A new online safe flight control method based on constrained reinforcement learning is proposed for the intelligent safety control of UAVs. This method adopts constrained policy optimization as the main reinforcement learning framework and develops a constrained policy optimization algorithm with extra safety budget, which introduces Lyapunov stability requirements and limits rudder deflection loss to ensure flight safety and improves the robustness of the controller. By efficiently interacting with the constructed simulation environment, a control law model for UAVs is trained. Subsequently, a condition-triggered meta-learning online learning method is used to adjust the control raw online ensuring successful attitude angle tracking. Simulation experimental results show that using online control laws to perform aircraft attitude angle control tasks has an overall score of 100 points. After introducing online learning, the adaptability of attitude control to comprehensive errors such as aerodynamic parameters and wind improved by 21% compared to offline learning. The control law can be learned online to adjust the control policy of UAVs, ensuring their safety and stability during flight.
BibTeX
@inproceedings{zhao2024online,
  author = {Jiawei Zhao and Haotian Xu and Zhaolei Wang and Tao Zhang},
  title = {{Online Safe Flight Control Method Based on Constraint Reinforcement Learning}},
  booktitle = {Drones},
  year = {2024},
  doi = {10.3390/drones8090429},
}
Figure