Intelligent Control of Morphing Aircraft Based on Soft Actor-Critic Algorithm

Shaojie Ma, Xuan Zhang, Yuhang Wang, Junpeng Hui, Zhu HanICCEIC 2022 · OpenReview

TL;DR

This paper tackles the challenge of controlling morphing aircraft, which exhibit highly nonlinear dynamics and uncertainties due to their changing shapes. They propose an intelligent control strategy using the Soft Actor-Critic (SAC) deep reinforcement learning algorithm, enhanced with network pre-training, to achieve stable flight control during deformation. The results demonstrate significant improvements in control accuracy and robustness against internal and external disturbances.

What Problem Does This Paper Address?

Morphing aircraft, capable of altering their aerodynamic shape in flight, offer enhanced performance but present a formidable control challenge. The continuous deformation leads to strongly nonlinear dynamics and significant uncertainties in the aircraft's model. Traditional control methods, like switching linear variable parameter robust control, often simplify the nonlinearities, while adaptive control methods can be overly dependent on model accuracy. Existing robust control methods, while effective, can be complex to design due to numerous parameters. This paper aims to address these issues by developing an intelligent control system that can adapt to the complex and uncertain dynamics of morphing aircraft.

Methods

The core of this paper's approach is the application of the Soft Actor-Critic (SAC) algorithm, a deep reinforcement learning technique known for its effectiveness in continuous control tasks. SAC utilizes an actor-critic framework with deep neural networks to represent the policy (actor) and value functions (critics). Key aspects of their methodology include:

  • State, Action, and Reward Design: The authors carefully define the state space (including altitude, velocity, angle of attack, pitch rate, and sweep angle), action space (elevator deflection and thrust), and a reward function designed to encourage accurate tracking of altitude and velocity while penalizing control effort and state divergence.
  • Network Pre-training: To overcome the initial instability and poor sample quality often encountered in deep reinforcement learning for complex systems, the paper employs network pre-training. A traditional controller's behavior is learned by a deep neural network, which then serves as the initial policy network for SAC. This significantly improves training efficiency and stability.
  • SAC Algorithm Details: The paper outlines the standard SAC update rules for critic and actor networks, incorporating entropy regularization to promote exploration. They also mention the use of two critic networks to reduce Q-function estimation errors and target networks for stability.

Datasets

Not discussed in the paper. The study relies on a simulated mathematical model of a variable sweep aircraft for training and evaluation.

Key Insights and Ideas

  • Reinforcement Learning for Morphing Aircraft: The paper demonstrates that deep reinforcement learning, specifically SAC, can effectively handle the complex, nonlinear, and uncertain dynamics of morphing aircraft, a domain traditionally addressed by more conventional control techniques.
  • The Power of Pre-training: Network pre-training is shown to be crucial for achieving stable and efficient training of the SAC agent. By initializing the policy network with a learned traditional controller, the agent starts from a more informed state, leading to better initial performance and faster convergence.
  • Robustness to Uncertainty: The proposed SAC-based controller exhibits strong robustness to both internal model uncertainties (due to deformation) and external disturbances (aerodynamic, structural, and density deviations). This is a critical finding for real-world applications of morphing aircraft.

Key Results

  • Improved Control Accuracy: Under nominal conditions, the SAC-optimized controller significantly reduced the integral absolute error for altitude and velocity tracking. The error for altitude dropped from 167.1219m to 7.4009m, and for velocity from 114.5735m/s to 7.0559m/s after optimization.
  • Robustness to Deviations: Even with 20% aerodynamic deviation, 15% structural deviation, and 10% density disturbances, the controller maintained stable flight. The integral absolute errors under these deviation states were 10.2310m for altitude and 7.9277m/s for velocity, indicating good performance despite significant uncertainties.
  • Training Performance: The cumulative reward generally increased during training, indicating that the SAC agent learned a progressively better control policy. While some fluctuations were observed due to SAC's exploration mechanism, the overall trend was positive.

Limitations and Future Work

  • Simulation-Based Verification: The authors acknowledge that the current verification is solely based on mathematical simulation. They emphasize that practical application verification is necessary to fully validate the proposed control strategy.
  • Complexity of Deformation: While the paper addresses deformation, the specific dynamics and control challenges associated with different types and extents of morphing might require further investigation.
  • Generalizability: The study focuses on a variable sweep UAV. Extending the approach to other types of morphing aircraft with different deformation mechanisms could be a future direction.

Summary updated

BibTeX
@inproceedings{ma2022intelligent,
  author = {Shaojie Ma and Xuan Zhang and Yuhang Wang and Junpeng Hui and Zhu Han},
  title = {{Intelligent Control of Morphing Aircraft Based on Soft Actor-Critic Algorithm}},
  booktitle = {ICCEIC},
  year = {2022},
  url = {https://openreview.net/forum?id=W7sT0FWr8a},
}
Figure