Abstract
SOREO (Safe and autOnomous dRones flEet navigatiOn) is a reinforcement learning-based system designed for autonomous drone fleet navigation in complex urban environments. By utilizing Q-learning to pre-calculate optimal, obstacle-free paths offline, the system enables safe and efficient package delivery without the need for expensive onboard sensors for real-time path optimization. This paper details the two-phase architecture—offline learning and online execution—and demonstrates its effectiveness in simulated grid environments with multiple drones and obstacles.
Introduction
While high-end military and commercial drones possess advanced autonomous capabilities, most consumer-grade drones remain limited in their ability to navigate complex environments. Coordinating a fleet of drones for tasks such as urban package delivery presents two primary challenges: identifying the shortest path to optimize battery life and time, and avoiding collisions with static obstacles like buildings. Traditional methods often require intensive real-time sensor processing or rigid formal models, which are difficult to maintain in evolving urban settings. SOREO aims to bridge this gap by creating a system that learns the geography of an area and coordinates multiple drones simultaneously.
System Architecture
The SOREO architecture is bifurcated into two distinct phases: an offline learning component and an online execution component.
Offline Component (The "Brain")
This phase employs Q-learning, a reinforcement learning algorithm, to map the environment. The system is provided with a virtual floor map, a set of permitted actions (eight-directional movement), and a reward function. Through iterative trial and error, the Q-learning processor identifies optimal transitions that avoid collisions and reach the target. These successful paths are stored in a "Routes Database."
Online Component (The "Pilot")
During active missions, the system queries the database for the optimal path. Navigation is managed by two specialized controllers:
- P-controller: Adjusts the drone's orientation to align with the next waypoint.
- Non-linear controller: Manages velocity, ensuring the drone decelerates appropriately as it approaches its destination to prevent overshooting.
Experimental Setup
The system was evaluated using custom-designed virtual environments representing geographic grids. These simulations tested varying levels of complexity, with environments containing 12, 18, and 24 obstacles.
The experiments involved fleets of 2 or 3 drones operating simultaneously, each assigned unique base and target coordinates. This setup allowed the researchers to assess the system's ability to handle multi-agent coordination within a shared airspace.
Key Results and Insights
The evaluation of SOREO yielded several critical insights:
- Sensor-Lite Optimization: By shifting pathfinding to an offline phase, SOREO eliminates the need for high-end onboard sensors for real-time optimization, allowing for simpler drone hardware.
- Robustness to Uncertainty: Q-learning effectively handles the stochastic nature of real-world robotics, where commands may not result in perfect state transitions.
- Scalability and Safety: The system consistently achieved a "zero collision" record and successfully managed a fleet of 3 drones in medium-sized areas with 18 obstacles.
- Modular Control: Decoupling route-finding from physical flight control makes the system more adaptable to different drone hardware.
Conclusion and Future Work
SOREO demonstrates a viable framework for RL-based fleet navigation. However, the authors identify several avenues for future research. The efficiency of the offline learning phase requires optimization to reduce pre-calculation time. Furthermore, while the current model assumes static obstacles, future iterations will investigate dynamic environments and moving targets. Finally, scaling the system to handle higher fleet densities (dozens or hundreds of drones) remains a key computational challenge.
Summary updated