SOREO: A System for Safe and Autonomous Drones Fleet Navigation with Reinforcement Learning

Réda Alami, Hakim Hacid, Lorenzo Bellone, Michal Barcis, Enrico Natalizio

Abstract

SOREO (Safe and autOnomous dRones flEet navigatiOn) is a reinforcement learning-based system designed for autonomous drone fleet navigation in complex urban environments. By utilizing Q-learning to pre-calculate optimal, obstacle-free paths offline, the system enables safe and efficient package delivery without the need for expensive onboard sensors for real-time path optimization. This paper details the two-phase architecture—offline learning and online execution—and demonstrates its effectiveness in simulated grid environments with multiple drones and obstacles.

Introduction

While high-end military and commercial drones possess advanced autonomous capabilities, most consumer-grade drones remain limited in their ability to navigate complex environments. Coordinating a fleet of drones for tasks such as urban package delivery presents two primary challenges: identifying the shortest path to optimize battery life and time, and avoiding collisions with static obstacles like buildings. Traditional methods often require intensive real-time sensor processing or rigid formal models, which are difficult to maintain in evolving urban settings. SOREO aims to bridge this gap by creating a system that learns the geography of an area and coordinates multiple drones simultaneously.

System Architecture

The SOREO architecture is bifurcated into two distinct phases: an offline learning component and an online execution component.

The SOREO system architecture, illustrating the interaction between the offline Q-learning component and the online navigation controllers.

Offline Component (The "Brain")

This phase employs Q-learning, a reinforcement learning algorithm, to map the environment. The system is provided with a virtual floor map, a set of permitted actions (eight-directional movement), and a reward function. Through iterative trial and error, the Q-learning processor identifies optimal transitions that avoid collisions and reach the target. These successful paths are stored in a "Routes Database."

Online Component (The "Pilot")

During active missions, the system queries the database for the optimal path. Navigation is managed by two specialized controllers:

  • P-controller: Adjusts the drone's orientation to align with the next waypoint.
  • Non-linear controller: Manages velocity, ensuring the drone decelerates appropriately as it approaches its destination to prevent overshooting.

Experimental Setup

The system was evaluated using custom-designed virtual environments representing geographic grids. These simulations tested varying levels of complexity, with environments containing 12, 18, and 24 obstacles.

A sample virtual environment grid used for testing, showing start points (green), target points (red), and static obstacles (black lines).

The experiments involved fleets of 2 or 3 drones operating simultaneously, each assigned unique base and target coordinates. This setup allowed the researchers to assess the system's ability to handle multi-agent coordination within a shared airspace.

Key Results and Insights

The evaluation of SOREO yielded several critical insights:

  • Sensor-Lite Optimization: By shifting pathfinding to an offline phase, SOREO eliminates the need for high-end onboard sensors for real-time optimization, allowing for simpler drone hardware.
  • Robustness to Uncertainty: Q-learning effectively handles the stochastic nature of real-world robotics, where commands may not result in perfect state transitions.
  • Scalability and Safety: The system consistently achieved a "zero collision" record and successfully managed a fleet of 3 drones in medium-sized areas with 18 obstacles.
  • Modular Control: Decoupling route-finding from physical flight control makes the system more adaptable to different drone hardware.

Conclusion and Future Work

SOREO demonstrates a viable framework for RL-based fleet navigation. However, the authors identify several avenues for future research. The efficiency of the offline learning phase requires optimization to reduce pre-calculation time. Furthermore, while the current model assumes static obstacles, future iterations will investigate dynamic environments and moving targets. Finally, scaling the system to handle higher fleet densities (dozens or hundreds of drones) remains a key computational challenge.

Summary updated

Abstract

This demonstration introduces SOREO, a system that explores the possibility of extending UAVs autonomy through machine learning. It brings a contribution to the following problem: Having a fleet of drones and a geographic area, how to learn the shortest paths between any point with regards to the base points for optimal and safe package delivery? Starting from a set of possible actions, a virtual design of a geographic location of interest, e.g., a city, and a reward value, SOREO is capable of learning not only how to prevent collisions with obstacles, e.g., walls and buildings, but also to find the shortest path between any two points, i.e., the base and the target. SOREO exploits based on the Q-learning algorithm.
BibTeX
@inproceedings{alami2023soreo,
  author = {Réda Alami and Hakim Hacid and Lorenzo Bellone and Michal Barcis and Enrico Natalizio},
  title = {{SOREO: A System for Safe and Autonomous Drones Fleet Navigation with Reinforcement Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.27058},
}
Figure