Adaptive Missile Avoidance Algorithm for UAV Based on Multi-Head Attention Mechanism and Dual Population Confrontation Game

Cheng Zhang, Junhao Song, Chengyang Tao, Zihan Su, Zhiqiang Xu, Weijia Feng, Zhaoxiang Zhang, Yuelei XuDrones 2025 · DOI

TL;DR

This paper introduces a novel reinforcement learning framework for UAV missile evasion that combines a risk-sensitive Proximal Policy Optimization (RPPO) algorithm with a multi-head attention mechanism and a dual-population adversarial training setup. This approach significantly improves the UAV's ability to adapt to diverse and unknown missile threats, achieving a 94% success rate in simulations.

What Problem Does This Paper Address?

Unmanned Aerial Vehicles (UAVs) are increasingly facing sophisticated missile threats, necessitating robust and adaptive evasion strategies. Traditional evasion methods often struggle with the dynamic nature of combat, the variability of missile guidance systems, and incomplete information. While reinforcement learning (RL) offers a promising avenue for dynamic strategy optimization, existing RL-based evasion algorithms often exhibit limitations in adapting to a wide range of unknown missile types, leading to poor generalization and potential overfitting to specific opponents. This paper aims to bridge this gap by developing an evasion strategy that is not only effective against known threats but also robust and adaptable to novel and diverse missile behaviors.

Methods

The core of this work lies in a novel training framework for UAV missile evasion. Key components include:

  • Risk-Sensitive Proximal Policy Optimization (RPPO): This algorithm is an extension of PPO that incorporates a risk-sensitivity hyperparameter. This allows the agent to learn strategies with different behavioral preferences based on varying risk levels, aiming for a more robust and adaptable policy.
  • Multi-Head Attention Mechanism: Integrated into both the actor and critic networks, this mechanism allows the UAV to selectively focus on historical state information. This is crucial for extracting latent features like missile guidance laws from sequences of observations, enabling a deeper understanding of the missile's behavior over time.
  • Dual-Population Adversarial Training: Instead of traditional self-play where an agent plays against itself, this approach pits a population of UAV evasion agents against a population of missile pursuit agents. This adversarial setup, combined with a RANK-based reward correction scheme, is designed to prevent cyclic counteractions and ensure policy diversity, leading to more generalized evasion strategies that can handle various missile types.
  • GRU Module: Used in conjunction with the multi-head attention mechanism, the GRU helps encode temporal information from the state sequences, further enhancing the agent's ability to understand dynamic missile behaviors.

Datasets

The paper does not explicitly mention using pre-existing datasets. Instead, it describes a simulation environment constructed based on the mathematical models of UAV, missile, and decoy dynamics. This environment is used for both training the proposed algorithm and for conducting extensive simulation experiments to evaluate its performance.

Key Insights and Ideas

  • The Power of Adversarial Diversity: The dual-population adversarial training framework is a significant departure from standard self-play. By training against a diverse set of missile agents, the UAV learns more generalized evasion tactics, avoiding the overfitting issues seen in methods trained against a single, static opponent.
  • Attention is All You Need (for Missile Guidance): The multi-head attention mechanism proves effective in allowing the UAV to infer complex missile guidance laws from observed motion patterns. This ability to "read" the missile's intent is critical for timely and effective evasion.
  • Risk Sensitivity for Robustness: Incorporating risk sensitivity into the RL algorithm allows for the generation of strategies that are not just optimal in expectation but also robust to uncertainty and potential failures, which is paramount in a high-stakes combat scenario.
  • Beyond Simple Evasion: The framework successfully integrates maneuver evasion with decoy deployment, demonstrating a sophisticated, multi-faceted approach to missile defense.

Key Results

  • Superior Survival Rate: The proposed method achieved a remarkable 94% successful evasion rate in comprehensive testing scenarios involving various missile types.
  • Faster Convergence: Comparative experiments showed that the proposed method achieved faster convergence to effective evasion strategies compared to basic PPO and MADDPG-based self-play methods.
  • Enhanced Adaptability: The trained evasion strategies demonstrated superior adaptability and generalization capabilities across different missile guidance laws and parameters, outperforming baseline methods when facing novel missile types.
  • Effective Decoy Deployment: Simulation samples illustrate how the UAV effectively uses decoy flares, particularly at longer distances, and then intensifies their use at closer ranges, successfully deceiving missiles.

Limitations and Future Work

  • Simulation vs. Real-World: The current study is based on a simulated environment. While the models are detailed, real-world deployment would require validation against actual flight data and potentially more complex environmental factors.
  • Computational Cost: While the paper mentions low computational overhead during inference, the training process, especially with dual-population adversarial training and multi-head attention, is likely computationally intensive.
  • Limited Scope of Threats: The current research focuses on missile evasion. Future work could explore extending this framework to handle more complex scenarios involving multiple threats, electronic warfare, and other adversarial actions.
  • Decoy Effectiveness: The paper implicitly assumes a certain effectiveness of decoy flares. Further research could investigate how to dynamically optimize decoy deployment strategies based on real-time assessments of their effectiveness against specific missile types.

Summary updated

Abstract

In recent years, UAVs have faced increasingly severe and diversified missile threats. To address the challenge that reinforcement learning-based missile evasion algorithms struggle to adapt to various unknown missile types, we introduce a risk-sensitive PPO algorithm and propose a training framework incorporating multi-head attention mechanisms and dual-population adversarial training. The multi-head attention mechanism enables the policy network to extract latent features such as missile guidance laws from state sequences, while the dual-population adversarial approach ensures policy diversity and robustness. Compared to conventional self-play methods and GRU-based evasion strategies, our method demonstrates superior training efficiency and generates evasion policies with better adaptability to different missile types.
BibTeX
@inproceedings{zhang2025adaptive,
  author = {Cheng Zhang and Junhao Song and Chengyang Tao and Zihan Su and Zhiqiang Xu and Weijia Feng and Zhaoxiang Zhang and Yuelei Xu},
  title = {{Adaptive Missile Avoidance Algorithm for UAV Based on Multi-Head Attention Mechanism and Dual Population Confrontation Game}},
  booktitle = {Drones},
  year = {2025},
  doi = {10.3390/drones9050382},
}
Figure