Situation-Aware Deep Reinforcement Learning for Autonomous Nonlinear Mobility Control in Cyber-Physical Loitering Munition Systems

Hyunsoo Lee, Soohyun Park, Won Joon Yun, Soyi Jung, Joongheon KimarXiv 2022 · arXiv

TL;DR

This paper introduces a situation-aware Deep Reinforcement Learning (DRL) framework designed to help "loitering munitions"—essentially suicide drones—navigate complex, obstacle-ridden battlefields. By combining the Deep Deterministic Policy Gradient (DDPG) algorithm with Unity-based "Raycast" sensing, the researchers developed a system that allows drones to autonomously adjust their flight paths nonlinearly to avoid buildings and reach targets with significantly higher success rates than traditional linear methods.

What Problem Does This Paper Address?

In modern urban warfare, drones are no longer just "professional toys"; they are critical tools for reconnaissance and combat. However, controlling them autonomously in a "loitering munition" capacity (where a drone loiters over an area before attacking a specific target) is incredibly difficult.

Most existing autonomous flight algorithms rely on linear trajectory control, which works fine in open skies but fails miserably in urban environments filled with "blockages" like buildings and structures. Furthermore, training these systems in the real world is impractical—gathering battlefield data is dangerous, and crashing expensive hardware during the learning phase is cost-prohibitive. There is a clear need for a robust, simulation-based approach that can handle the nonlinear, continuous movements required to dodge obstacles in real-time.

Fig. 1: The use of drones in the military battlefield, i.e., autonomous drone mobility control for loitering munition.

Methods

The authors propose a Cyber-Physical System (CPS) approach, using the Unity 3D engine to create a high-fidelity virtual battlefield. Their method relies on two key technical pillars:

  1. Deep Deterministic Policy Gradient (DDPG): Unlike standard Deep Q-Networks (DQN) that handle discrete actions (e.g., "move left" or "move right"), DDPG allows for a continuous action space. This is vital for drones, as it enables smooth, nonlinear adjustments in 3D space ($x, y, z$ directions) rather than jerky, step-based movements.
  2. Raycast-based Situation Awareness: To give the drone "eyes," the researchers utilized Unity’s Raycast function. This shoots virtual lasers in various directions to detect the distance to nearby obstacles. This information is fed directly into the DRL state space.

The drone is trained with a reward function that penalizes it for being far from the target or colliding with obstacles (using a negative reward inversely proportional to the distance from an object) and rewards it for successfully reaching the target (a tank asset).

Fig. 3: Raycast information for situation-awareness which is essentially required for autonomous nonlinear drone mobility control.

Datasets

The study does not use a static external dataset. Instead, it utilizes a dynamic Unity 3D environment to generate training data. The researchers built a custom "city environment" and evaluated their algorithm across a spectrum of difficulty levels by varying obstacle density from 0% to 100%. This allowed them to benchmark the drone's robustness in everything from open fields to dense urban "canyons."

Key Insights and Ideas

  • Continuous vs. Discrete Control: The researchers found that DDPG significantly outperformed Advantage Actor-Critic (A2C) algorithms. Because A2C often struggles with continuous behavior modeling, its "score" (reward accumulation) remained much lower and less stable than DDPG's during the learning steps.
  • The Power of "Virtual Lasers": Integrating Raycast data into the DRL state space transformed the drone from a blind path-follower into a situation-aware agent. By receiving negative rewards before a collision (based on Raycast distance), the agent learned proactive avoidance rather than just reactive recovery.
  • Nonlinear Superiority: Traditional Linear Mobility Control (LMC) assumes a clear path. In complex environments, the "shortest path" is rarely a straight line. The DDPG agent learned to adjust altitude and lateral position simultaneously to "weave" through buildings.
Fig. 2: RL score plotting (i.e., reward value dynamics) for each step.

Key Results

The proposed nonlinear control algorithm was compared against a baseline Linear Mobility Control (LMC) model across various obstacle densities:

  • Low Density (10-30%): The proposed model showed a modest improvement (roughly 9% to 45% better than LMC).
  • High Density (70-100%): This is where the algorithm shined. At 90% obstacle density, the proposed model achieved an average of 7.7 successful attacks per 20 trials, while the LMC model managed only 1.7. This represents a 352.94% performance improvement.
  • Robustness: Even at 100% obstacle density—a "harsh" environment where LMC is essentially unusable—the situation-aware DRL agent maintained a success rate of over 35%, proving its resilience in extreme conditions.

Limitations and Future Work

While the results in the Unity simulation are promising, the authors acknowledge the "Sim-to-Real" gap. Real-world drones face physical variables like wind, sensor noise, and battery fluctuations that are difficult to model perfectly in a virtual environment.

The authors suggest that future research should focus on implementing this algorithm on embedded drone platforms (actual hardware) equipped with physical sensors (like LiDAR or ultrasonic sensors) that can replicate the Raycast functionality in the real world. Additionally, exploring multi-drone swarm coordination using similar nonlinear control remains an open area for investigation.

Summary updated

Abstract

According to the rapid development of drone technologies, drones are widely used in many applications including military domains. In this paper, a novel situation-aware DRL- based autonomous nonlinear drone mobility control algorithm in cyber-physical loitering munition applications. On the battlefield, the design of DRL-based autonomous control algorithm is not straightforward because real-world data gathering is generally not available. Therefore, the approach in this paper is that cyber-physical virtual environment is constructed with Unity environment. Based on the virtual cyber-physical battlefield scenarios, a DRL-based automated nonlinear drone mobility control algorithm can be designed, evaluated, and visualized. Moreover, many obstacles exist which is harmful for linear trajectory control in real-world battlefield scenarios. Thus, our proposed autonomous nonlinear drone mobility control algorithm utilizes situation-aware components those are implemented with a Raycast function in Unity virtual scenarios. Based on the gathered situation-aware information, the drone can autonomously and nonlinearly adjust its trajectory during flight. Therefore, this approach is obviously beneficial for avoiding obstacles in obstacle-deployed battlefields. Our visualization-based performance evaluation shows that the proposed algorithm is superior from the other linear mobility control algorithms.
BibTeX
@misc{lee2022situation,
  author = {Hyunsoo Lee and Soohyun Park and Won Joon Yun and Soyi Jung and Joongheon Kim},
  title = {{Situation-Aware Deep Reinforcement Learning for Autonomous Nonlinear Mobility Control in Cyber-Physical Loitering Munition Systems}},
  year = {2022},
  eprint = {2301.00124},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2301.00124},
}
Figure